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(54) A gradient based method tor providing values for unknown pixels In a digital Image 

(57) A method of correcting one or more defect pix- 
els in a defect region of a source image, which image 
has both the defect pixels and non-defect pixels each of 
which is represented by at least one defect pixel signal 
and non-defect pixel signal, respectively. The method 
comprises: 



a) searching for edges in a non-defect region of the 
image adjacent to the defect region by: (i) compar- 
ing relative pixel parameters in the non-defect 
region with a predetermined tolerance to locate a 
sequence of pixels representing an edge segment 
the members of which have relative parameters 
within the tolerance and which sequence is within a 
predetermined distance of the defect region; and (ii) 
comparing the length of an edge segment with a 
minimum length to Identify as an edge a segment 
with at least the minimum length; 

b) generating at least one pixel signal for the defect 
region based on the signal of at least one non- 
defect pixel lying in a direction from the defect pixel 
which is determined by the direction of an edge. 

An apparatus for performing the method and a 
computer usable medium containing computer code tor 
performing the method, are also provided. 
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Description 

Field at the Invention 

5 This Invention relates to a method, and apparatus for performing the method of correcting one or mors delect pixels 
in a defect region of a digital image signal representing a source image. 

Bflekarounderftha Invention 

10 Conversion of analog images into digital data has become widespread for a variety of applications, including stor- 
ing, manipulating, transmitting and displaying or printing copies of the Images. For example, sequences of frames from 
a motion picture film are often digitized, image enhancement or special effects are performed on the digitized images 
and the result written onto a blankfilm. Also, images captured on photographic media are being converted to digital data 
and stored on compact discs for readout and display as a video image or lor printing with various types of color printers. 

is In order to capture the photographic image digitally, the film frame is scanned with a light beam, and the light transmitted 
through the film is detected, typically as three primary color light intensity signals, and digitized. The digitized values 
may be formatted to a standard for video display and stored on compact disc or magnetic media. Such film digitizers 
take a variety of forms and the various common aspects of film digitizing, particularly line illumination and linear CCD- 
based digitizers, are described in greater detail in commonly assigned US. Patent No. 5,012,346. Also photographic 

20 prints can be digitized using reflection scanners. 

Providing new pixel values for corrupted or defect pixels is a common operation in digital image processing. Part of 
the image can be damaged by scratches or degraded by dust Corrupted pixel values do not represent original image 
information and hence these values cannot be used for restoration purposes. New pixel values must be supplied con- 
sistent with the image content surrounding the area to be replaced. This process is referred to here as a pixeHill. 

ss Another example of an image processing operation requiring pixelf ill is an object removal from the image, far example 
rig and wire removal In motion pictures. 

Simple nondirectional averaging or erosion techniques are usually used to fill small, compact defect regions in a 
film frame. These methods produce undesired smoothness in the replaced regions and severely distort edges that are 
intersected by the regions to be filled. The problem of reconstructing edges is especially important for long, narrow 

30 regions to be filled due to the high probability they intersect multiple objects in the image. This problem is addressed in 
the method described by Massmann et al. in U.S. Patent Number 5,097,521 . The above method has some disadvan- 
tages. First It assumes essentially vertical regions to be filled. It may also not handle well the situation where multiple 
edges intersect in the defect region. 

Kwoket al. in US Patent Number 5,097,521 disclose an error concealment system based on the gradient field. In 

3s this system a two dimensional block of defect data is filled. This system may not handle well the case where objects 
crossing the defect block have a width much smaller than the size of the defect block. 

Previous methods can produce artifacts when curved edges, object corners or T-type" edge intersections are cov- 
ered by the defect region. 

It would ba desirable then, to have a method of providing corrected pixal values for pixels in a defect region, which 
40 can produce reconstructed image content in the defect region which is fairly consistent with the image content of the 
surrounding area, even in the case where the defect region intersects objects with a width smaller than, or comparable 
to, the width of the defect region or where multiple edges Intersect In the defect region. 

Summary of the Invention 

48 

The present Invention then, provides a method of correcting one or more defect pixels In a defect region of a source 
image. The source image has both defect pixels and non-defect pixels each of which is represented by at least one 
defect pixel signal and non^efect pixel signal, respectively. The Invention assumes the presence of an input digital 
image along with an indication of defect regions, such as a mask image. Pixels of a given value (e.g. equal 0) in the 
so mask image define defect pixels in the input image. In one aspect, the method comprises: 

a) searching a non-defect region of the image adjacent to the defect region for sequences of pixels representing 
edge segments by: (i) comparing relative pixel parameters in the hon-defact region with a predetermined tolerance 
to locate a sequence of pixels representing an edge segment, the members of which have relative parameters 

ss within the tolerance and which sequence is within a predetermined distance off the defeat region; and (ii) comparing 
the length of an edge segment with a minimum length to identify as an edge a segment with at least the minimum 
length; 

b) generating at least one pixel signal for the defect region based on the signal of at least one non-defect pixel lying 
in a direction from the defect pixel which is determined by the direction of an edge. 



EP0 778 543A2 

The method of the present Invention particularly land* Itself to Implementation on a computer, preferably a pro- 
grammable digital computar. Tha necessary calculations (e g. generating gradients, comparing lengths, generating 
pixal signals) described below can be performed by the computer's processor, while anything requiring storage can be 
stored in a storage device (e.g. RAM or disk) In the computer. 

5 The relative pixel parameters used for detecting edges will generally Include a parameter which directly defines an 
edge shape: For example, an edge directional angle can be used as tha parameter, where the edge directions! angle 
of a pixel is taken as being perpendicular to the gradient direction at that pixel. One way of locating edges is to select 
the non-defect region as a "eliding" window d limited dimensions adjacent to the defied region, which window defines 
a local non-defect region. An edge in that window can then be located using a gradient field, as described in more detail 

10 below, ft will be noted that by "adjacent" does not necessarily mean immediately adjacent in that there may be some 
preselected distance separating the center of the window or end of an edge from the defect area. In particular, as 
described below, a defect extension distance due to blur end gradient operations, separates the center of the sliding 
window from the defect region. 

Only the directional attribute of pixel gradients may be compared (although preferably both directional angle and 

;5 magnitude are used). The method will locate a sequence of pixels which extends in at least one direction from the start 
pixal (and including the start pixel itself), tha members of the located sequence having relative gradient attributes at 
least one of which attributes is within a predetermined tolerance and which sequence is within a predetermined dis- 
tance of the defect region. The length of an edge segment is compared with a minimum length and, if the minimum 
length is met, the segment is classified as an edge. The window can then be moved, as described in more detail below 

20 and the process repeated so that an entire area adjacent to and about the non-defect region has been searched for 
edges by means of searching such local non-defect regions in sequence. 

Using the sliding window method as described, an area adjacent to and around the defect region can be treated as 
a plurality of local non-defect regions. However, non-defect regions adjacent to and around the defect region could be 
searched for edges in a similar manner without necessarily using the preferred sliding window method. 

6 Pixel signals for the defect region can be generated based on the edge that extends into the defect region. In this 
case, a model of edge shape would be determined which would then be used to predict the shape of the edge extended 
into the defect region. For example, "for a curved edge a model of the curvature coufd be developed (for example, an 
equation representing the curvature) which is used to predict the location of the curve in the defect region. Values of 
pixels on the defect portion of the edge could then be generated for the predicted defect portion of the edge based on 

so the signals of at least some of the pixels of the edge (such as by nearest neighbor or other extrapolation or interpolation 
techniques). Values for other pixels in the defect region not lying on an edge (which will typically be within a predeter- 
mined distance ol an edge) can also be generated based on the signal of at least one non-defect pixel lying In a direc- 
tion from the defect pixel which is determined by the direction of an edge. 

However, in one embodiment of the present invention pixel signals for the defect region may be generated when 
35 two edges are found in the local neighborhood of the defect region. In this case, pixel parameters of the ends erf those 
edges adjacent to the defect region are compared with at least one predetermined tolerance (a predetermined toler- 
ance for both gradient magnitude and gradient directional angle can be used). For example, the gradient magnitude and 
the local gradient directional angle of the pixels of the ends can be compared with a predetermined acceptable magni- 
tude variation and relative angle range. In this case a model of edge shape would be determined which would then be 
40 used to predict the shape of the edge segment that connects two edges in the defect region. For example, an edge seg- 
ment connecting two edges in the defect region can be modeled by a non-linear function to represent a curvature. Pixel 
signals can then be generated for defect region pixels on the location of an edge segment connecting the two ends In 
the defect region. The generated signals are based on at least one pixel signal of the two detected edges. In this regard, 
pixel signals may be generated for some or all of the pixels of the edge segment connecting two edges through the 
45 defect region. Values for other pixels in the defect region not lying on the edge segment connecting two edges (which 
will typically be within a predetermined distance of the edge segment) can also be generated based on the signal of at 
least one non-defect pixel lying in a direction from the defect pixel which is determined by the direction of at least one 
edge and/or the edge segment connecting the two edges. 

It will be appreciated that in the preceding embodiment, it is not necessary that the two edges are both first identi- 
fy fied. For example, one edge could be first identified by using gradient field as described. Then, the location of a line 
which extends from the edge and through the defect region, is determined, This location can be based on the direction 
of at least a portion of the edge (such as the end of the edge adjacent to the defect region}. Then a non-defect region 
adjacent to the defect area and opposite the edge, is searched tor a sequence of pixels representing an edge segment 
extending from a first non-defect pixel intersected by the line extension of the edge through the defect region. This can 
& be done in a manner already described by comparing the difference in pixel parameters (such as edge directional 
angles cf the pixels) with the predetermined tolerance- 
It will be appreciated that In the method of the present Invention, additional tests can be applied to decide when an 
edge (typically a connected edge, as described below) is found. For example, a parameter representative of pixel sig- 
nals in a region adjacent to at least a first side of one of the edges, can be compared with the same parameter repre- 
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sentatlv* of pixel signals In a region adjacent to a corresponding first side of the other edge. Preferably the parameter 
for pixel signals in a region adjacent to a corresponding second side of both edges is also tested against the predeter- 
mined tolerance and must fall within it, before pixel signals are generated between the two ends. Such a parameter can, 
for example, represent average pixel values for each color channel (such as RGB, CMY, monochrome and the like) in 

5 the compared regions, In this case, this additional test can be regarded as a color match constraint to be met by the 
edges. Variance of the pixel values for each region can also be included in the parameter. As to which sides are "cor- 
responding" sides, this will depend on the complexity of shapes formed by multiple edges which are to be handled by 
the method. For example, where the method is to handle single edgss which may traverse the defect region [i.e. single 
line case), "corresponding" sides would be the sides facing in the same direction perpendicular to the edge. If the 

to method is to handle other or more complex shapes formed by edges, the corresponding sides would depend on the 
type of shape to be handled (examples of these are provided below). 

In the preferred implementation of the present invention, the method comprises searching a non-defect region adja- 
cent to the defect region, for edge segments, using the methods (preferably gradient based) already described or any 
other suitable method, and comparing the length of edge segments with a predetermined minimum length to identify as 

is edges, those segments of at least the minimum length. The locations of these detected edges and relevant information 
are stored, for example in a computer random access memory (RAM), end define an edge map. This edge map i& then 
searched for geometric primitives. A "geometric primitive" is a model of a preselected arrangement of one or more 
edges. Examples of geometric primitives includes corners, T-junctions and X-junctions. This searching step may be 
accomplished by comparing at least one relative geometric parameter of a set of at least two edges with a geometric 

20 constraint which corresponds to a geometric primitive. For axanple, an angle of intersection of two edges can be com- 
pared with a constraint corresponding to a range of intersection angles. When the set of edges meets the geometric 
constraint, plus any other imposed constraint (such as the color match constraint described below), it is considered that 
a geometric primitive has been delected and the edges of the set meeting the constraints are parte of a "detected" geo- 
metric primitive. 

a? A detected geometric primitive can be extracted by extending or reducing (that is, removing portions of an edge), 
In the map, one or more edges of the detected primitive so that they Intersect In the same pattern as defined by the 
primitive. Alternatively, the primitive can be extracted by replacing, in the maps the edges detected as a primitive with 
one or more new edges representing the primitive. In the first case, for edges of a detected primitive which intersect the 
defect region, they will be extended into the defect region typically based on their directions as they enter the defect 

30 region. This can be accomplished in different ways. Far example, in a preferred way this is accomplished by extending, 
in the map, all detected edges intersecting tha defect region through the defect region. This is -followed by removing por- 
tions of those edges which are pahs of a detected primitive which portions extend beyond Intersection points of the 
edges matching the geometric configuration of the detected primitive. In a second way, this can be accomplished by 
extending only those edges which are parts of a detected primitive in a way matching the geometric configuration of the 

35 detected primitive. For edges of a primitive intersecting outside the defect region, it may be necessary to reduce one or 
more of them by removing portions extending beyond an Intersection point of them. 

To avoid long processing times in a computer implementing the present method, it will be apparent that the models 
of supported geometric primitives will be those arrangements of edgee (whether straight, curved or other shapes) con- 
sidered to most likely be encountered in images to be processed by the present method. 

40 One particular geometric parameter which may be used is the angle of intersection of two edges in the edge map. 
The corresponding geometric constraint would be an allowable range of intersection angles of two edges. The "inter- 
section angle" Is the angle between the directions of two edges at the pixels where they actually Intersect (In the case 
where they intersect outside the defect region) or between their directions at the pixels where they enter the defect 
region (in the case where the edges do not intersect outside the defect region but their directions intersect inside the 

4? defect region). The edge direction at a given point is defined by an average edge directional angle of a selected set of 
ac(Jacent edge points. 

While meeting one or more geometric constraints is a requirement -for edges being part of a detected geometric 
primitive, it may not be the only requirement. It is preferred that an additional constraint of a parameter representative 
of pixel signals in regions adjacent toone side, preferably two sides, of ai least two edgee must also be met. Particularly, 
so this would be a color match constraint in the case of color images. 

Preferably only those edges in the edge map which are within a predetermined maximum distance of the defect 
region are searched for geometric primitives, 

In the described preferred implementation, signals for defect region pixels are generated each of which is based on 
the signal of at least one ncn^defect pixel lying on a predictor line having a direction based on a part of a local detected 
55 geometric primitive. The predictor line will generally be parallel to a line representing a part of a detected geometric 
primitive. In the case of a detected line primitive though, it may be in a direction parallel to a line positioned in the edge 
map In what Is considered the best average position between two edges detected as a line primitive. 

Preferably, if there ie more than one part of at least one geometric primitive within a predetermined maximum dis- 
tance of a defect pixel, then estimated pixel signals for the defect pixel are calculated based on at least one non-defect 
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pixel lying on each of the predictor lines passing through the defect pixel, and the final pixel signal [ft a weighted sum of 
estimated pixel signals. Each estimated pixel signal is weighted in an inverse relationship to the distance between pre- 
dictor line and the corresponding geometric primitive part, as further described below. 

In an alternative method of the present invention for correcting defect pixels (referred to as a "modified pinwheel 

5 method 11 ) which are within a predetermined distance of at least one dominant direction of the gradient field, a plurality 
of line segments through a selected defect pixel are allocated. Each of the line segments is composed of the defect pixel 
and non-defect pixels about said selected defect pixel. At least one representative model of the non-defect pixel signals 
is determined along each said line segment. The deviation of the non-defect pixel signals along each said line segment 
from the model is determined, as is atleast one dominant direction of the local (by local is meant within a predetermined 

w distance) gradient field. For a dominant direction there must be a preselected minimum number of pixels with 1he gra- 
dient magnitude above a specified threshold and with gradient directional angles within a predetermined tolerance of 
this direction. At least one line segment is selected based on a figure of merit. The figure of merit is a value which 
increases with decreased deviation of the non-defect pixels along the selected line segment, and increases with the 
alignment of the line segment with the dominant directions of the local gradient field (which may, for example, represent 

15 an edge). Figures of merit and their calculation are discussed further below. An estimated pixel signal tor the defect 
pixel is calculated based on at least one non-defact pixel for each selected line segment and the final pixel signal is a 
weighted sum of estimated pixel signals with weights proportional to figures of merit of the selected line segments. 

The modified pinwheel method in a preferred implementation is used as an additional step for obtaining corrected 
signals for defect pixels which are beyond a threshold distance of any part of a detected geometric primitive or there are 

20 no detected geometric primitives, in this implementation the selected line segments do not intersect any detected geo- 
metric primitives and the direction of at least one local edge not detected as a part of any geometric primitive is used 
as the dominant direction of the local gradient field. 

The present invention further provides an apparatus for executing the method of the invention. The apparatus is 
preferably a programmed digital computer with a suitable memory (e.g. RAM, magnetic or optical disk drive, or other 

2$ suitable storage means) for storing the digital image signal. A computer program product is also provided, which has a 
computer readable storage mBdlum with executable coda for performing each of the steps of the method of the present 
invention. The computer readable storage medium way comprise, for example: magnetic storage media such as mag- 
netic disc (such as a floppy disc) or magnetic tape; optical storage media such as optical disc, optical tape, or machine 
readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory 

30 (ROM); or any other physical device or medium employed to store a computer program. 

The present invention then, provides a method and apparatus for providing corrected pixel values for pixels in a 
defect region of an Image signal, The Invention can produce reconstructed Image content In the delect region which Is 
fairly consistent with the image content of the surrounding area. This is the case even where the defect region intersects 
objects where the object width is smaller then, or comparable to, the width of the defect region or where multiple edges 

35 intersect in the defect region. 

Brief Description of the Drawing s 

Figure 1 is a block diagram of an apparatus of the present invention; 
40 Figure 2(a) is a pictorial view of a digital image with a defect pixel region ; 

Figure 2(b) is a pictorial view of a defect map for the digital image in Figure 2(a); 

Figure 2(c) Is a pictorial view of a corrected digital Image after correction of the defect pixels In the digital Image In 
Figure 2(a); 

Figure SA is a block diagram broadly illustrating the method of the present invention as edge detection, edge 
is manipulation and correction processes. 

Figure 3B Is a pictorial representation of detected edges, 

Figure 4 is a flow chart of the edge detection process. 

Figure 5A is a flow chart of the edge thinning process. 

Figure 55 is a flow chart of the geometric primitive extraction process. 
so Figures 6A - 6F is a pictorial representation of geometric primitives. 

Figure 7 is a flow chart of the correction process. 

Figures 8A - 8F is a pictorial representation of line segment predictors used in the process of estimating corrected 
values of defect region pixels. 

Figure 9 is a flow chart illustrating a "standard pinwheel" method for correcting defect pixels in a digital image in 
55 accordance with the present invention; 

Figure 10 is a flow chart illustrating a process for estimating corrected values for a defect pixel in accordance with 
the standard pinwheel method; 

Figure 1 1 is a flow chart of another embodiment of the standard pinwheel method illustrating a process tor estimat- 
ing corrected values for a defect pixel; 
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Figure 12 Is allow chart Illustrating an alternative embodiment In the standard plnwheel method for defining the line 
segments in the process for estimating corrected values for a defect pixel in Figures ID and 1 1 ; 
Figure 13 is a pictorial view of the digital image of Figure 2(a) illustrating a line segment being defined as described 
in Figure 12; and 

5 Figure 14 Is a pictorial view of an example Illustrating line segments defined about a defect pixel. 

Embodiments of the Invention 

An apparatus 8 in accordance with the present invention includes a programmed digital computer system 10 cou- 
io pied to an image reader 20, an imsga output device 24, and an user interface 22. Computer system 1 0 operates accord- 
ing to a set of Instructions to correct digital Images by providing new values for defect pixels within a defect region In the 
digital image*. With the apparatus and method, defect regions in digital images can be corrected so that regions are 
entirely unnoticeable to a viewer or less visually discernible because values for defect pixels have been estimated which 
are consistent with the overall image. 
is Referring more specifically to Figure 1, computer system 10 includes a processor 12, a memory 14 and a hard disk 
1 8, which are coupled together with an internal bus 1 8 for communicating both data and addresses between these com- 
ponents. Preferably, digital images are stored on hard disk 18 in system 10. 

Image reader 20 is coupled to computer system 10, and provides a digital image to the computer system 10 via 
internal bus 16. This digital image is stored in memory 14. In this particular embodiment, image reader 20 is a digital 
30 scanner, such as Kodak Professional Photo-CD 4045 Film Scanner, or digital camera, although any type of devioe for 
providing a digital image may be used. The digital image captured by reader 20 is the digital representation of any hard 
copy image such as frames of a motion picture film. Computer system 10 may also receive digital images from other 
sources, such asdigital images stored in hard disk 18 Each digital image is composed of an array of pixel values having 
one or more color channels. 

ss Image output device 24 is coupled to computer system 1 0. In this particular embodiment, output device 24 may be 
any type of digital Image output device, such as a video display, a printer, an external non-volatile memory storage 
device, or a transmission line to other computers within a network. Images stared in memory 14 may be output 1o image 
output device 24 through internal bus 16. In the case of source images representing frames from a motion picture, the 
images following correction by the present apparatus and method can, for example, be printed onto a film. 

3D User interface 22, such as a keyboard or mouse device, is coupled to computer system 1 0 and allows for a user to 
control and interact with apparatus 8, such as building of a defect map of the digital image stored in memory 14. User 
Inputs through user Interface 22 are described later, 

Referring to Figures 2(a-c) and 3, the operation of apparatus B will be discussed. When computer system 10 is 
started, system 10 can receive a digital image from image reader 20 and store the image in memory 14. Alternatively, 

35 the digital image can be transferred from hard disk 18 to memory 14. The image stored in memory 14 will hereinafter 
be referred to as the "source Image, 1 An example of the source image 25 Is shown In Figure 2(a) with defect region 26. 
The source image maybe comprised of a single color channel (monochrome], or have multiple color channels, such as 
RGB. 

Once the source imaga is stored in memory 14, then the defect regions in thesource image are located. Preferably, 

*? this is performed by the user identifying the approximate image areas containing the defect regions through user inter- 
face 22, and then processor 1 2 automatically detecting the defect pixels in those areas identified by the user. One sys- 
tem for Identifying defect pixels Is a system disclosed In EP 0 624 846 A3. Briefly, this defect pixel Identification system 
automatically identifies small local regions of an image which are anomalous in both brightness/color, local contrast, 
and size, and outputs a map of the defect pixels in the image. Alternatively the defect pixele may be selected manually 

*g by the user, which can be performed by a user generating a digital mask via user interface 22 (for example, by a graphic 
"painting" software application) Identifying the defect pixels, 

Once the defect regions are located, a defeat map is built and stored in memory 14 (step 104). In the defect map, 
pixels with a specified value correspond to defect pixels in the source image. One example of a defect map is shown in 
Figure 2(b) for the defect region 26 of the source image 25 in Figure 2(a). In this axarrple. defect pixels have values of 

so 0 (black), and non-defect pixels have values of 255 (white). 

The method of the present invention, as implemented using the foregoing apparatus, will now be described. Figure 
3 is a block diagram of a correction method of the present invention (sometimes referenced as a "pixelfill process"), The 
pixelfill of defect regions can be considered in terms of the three following steps: an edge detection step 102, an edge 
manipulation step 104, end a correction step 106. In the detection step 102, strong edges that intersect defect regions 

55 are detected. The detected edges are stored in the edge map. The edges in the edge map are processed in the edge 
manipulation step 104 (e.g. edge thinning, geometric primitive extraction). Third, in the correction step 1 06, defect pixels 
are provided with new values based on the local non-defect pixels, An edge map Is used to define the set of non-defect 
pixels used for the correction of a given defect pixel. These steps are described in more detail below. 
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1 . The edge detection step, 

The edge detection step 102 is based on the gradient field- The gradient field has two attributes: magnitude and 
directional angle. 

5 The local gradient field Is calculated for non-defect regions adjacent to defect regions. One criterion for selecting 
the set of non-defect pixals far gradient calculation is that included non-defect pixels must be within a given distance 
from the defect region. A suggested distance value is three times the maximum width of the defect region. Width of the 
defect region fit any defect pixel refers to the minimum width across the defect region at that pixel. Different embodi- 
ments may choose different ways of calculating the gradient field. A preferred implementation for color imagery is to Cai- 
ro culata the vector gradient. In this case the image is considered as a two dimensional vector field with the number of 
attributes corresponding to the number of channels In the Image, The magnitude of the gradient Is the square root of 
the largaet eigenvalue of D T D. The direction of the gradient is the corresponding eigenvector. D is a spatial partial deriv- 
atives matrix. D T is a transposed matrix D. Spatial partial derivatives can be calculated using a 3x3 Prewitt operator. 
Another possible implementation calculates gradient field based on scalar gradients of each channel. The gradient 

15 magnitude is then the square root of the sum of the squares of the gradient magnitudes of each color channel. The gra- 
dient directional angle can then be calculated as a weighted sum of directional angles of each channel. 

A smoothing operation should be performed prior to the gradient field calculation for the images with a significant 
level of noise. A uniform blur was used for testing the preferred embodiment of the invention. The size of the uniform 
blur depends on the noise level of the image. 

20 Figure 3B illustrates the detection of edges in the edge detection step. An actual defect region D represents the 
defect pixels of unknown values. The extended delect region, designated by ED, includes actual defect region D plus 
adjacent non-defect pixels where the gradient is not considered valid due to the proximity of the defect pixels affecting 
blur and gradient operations. The extended defect region ED extends beyond the actual defect region D by a distance 
DD where DD - g + b , where 2g + 1 is the gradient operator size and 2b + 1 is the blur operator size. 

£S Edge detection begins with finding a local gradient maximum, as described particularly in connection with Figure 
4. Next, an edge expansion process starts from the local gradient maximum point S to locate a sequence of pixels rep- 
resenting an edge segment. In the edge expansion process the next edge point is selected as the closest point to the 
current edge point in the edge direction. The edge direction is calculated for each edge point based on the local gradient 
directional angles. First, the edge is extended toward the defect region. This stage is called forward edge expansion. 

so The extended defect region ED is entered at the point EN. ff the number of edge points is less than a specified threshold 
number LMIN, representing a required minimum adga langth, then the expansion is continued in a second stage from 
the start point S In the opposite direction until the required minimum length LMIN Is reached at the point R This second 
stage of the edge expansion is called a reverse edge expansion. Point F is referred to as the first point of the edge and 
the edge part extending between point F and point EN is referred to as the delect entry part of the edge. LMIN will be 

35 specified by a user dependent on the type of source images. When the extended defect region is entered in the forward 
edge expansion process and the number of edge points Is greater than or equal to LMIN, the reverse edge expansion 
is not performed and point S is the first point of the edge. Next, the edge is extended from the defect entry point EN 
through the defect region to the defect exit point EX in the direction determined by the edge directional angles of the 
edge points preceding the defect entry point EN. The line segmant between EN and EX is referred to as the defect part 

40 of the edge, and the angle defining the direction of the defect part of the edge is referred to as the defect entry angle. 
The edge expansion is continued from the point EX to point Lfor a specified number of points equal to the defect entry 
part length. Point L Is referred to as the last point of the edge. The edge part extending between point EX and point L 
is referred to as the defect exit part of the edge. 

Each point of the defect entry and defect exit part of the edge must satisfy magnitude and angle conditions. The 

*s magnitude condition means the gradient magnitude at the edge point must be greater than or equal to a specified 
threshold MMIN. The angle condition means the edge directional angle difference between two consecutive edge 
points is smaller than a specified threshold AM AX. If either of the magnitude or angle conditions is not satisfied the 
expansion process stops. AMAX and MMIN are defined by a user depending on the source image. In the preferred 
implementation AMAX is in the range of 15 to 22° and MMIN is in the range of 15% to 60% of the maximum gradient 

60 magnitude. Alternatively, these parameters can be defined automatically based on the noise in the image and the max- 
imum gradient magnitude. 

Two types of edges are detected; connected and not connected. The connected edge consists of three parts; a 
defect entry part (the edge part between points F and EN) , a defect part (the edge part between points EN and EX) and 
a defect axrt part (the edge part between points EX and L). The connected edge must satisfy magnitude and ancle con- 
55 ditions (both specified above) for each point of defect entry and dafact exit part, minimum length requirements and a 
color match constraint. The minimum length requirement? mean the defect entry part of the edge must have at least 
LMIN length and the defect exit part of the edge must have at least a maximum of 3 pixels and 1/2 of LM IN. The color 
match constraint requires that there should be a color match between areas A0.A2 (which are areas adjacent to one 
side of a connected edge) and A1 , A3 (which are areas adjacent to the second side of a connected edgs). These areas 
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extend a given distance perpendicular to the defect entry and defect exit parte of the edge. In the preferred Implemen- 
tation the area extension distance is equal to the half defect width. There is a color match between two areas if a Stu- 
dent's t-test is passed for a specified confidence level. The confidence level will be specified by the user depending 
upon the source image. Preferably the confidence level is in the range of 0.3 to 0.7. 

The not connected edge consists only of two parts: a defect entry part (the edge part between points F1 and EN1) 
and a defect part (the edge part between points EN1 and EX1). The defect exit point EX1 is the last point of the not 
connected edge. The not connected edge must satiety magnitude and angle conditions (both specified above) for each 
point of the defect entry part and minimum length requirement. The minimum length requirement means the defect 
entry part has at least LMIN length. 

Both the connected and not connected edges are stored in an edge map. The edge map allows finding detected 
edges at any pixel of the source Image and retrieving edge attributes such as edge type (connected, not connected), 
edge length, color match attributes, all edge points, and the like. 

Figure 4 shows a flow chart of the process of searching for edges, as implemented by the appropriate software 
commands in the computer system of Figure 1 . The non-defect region adjacent to the delect region is searched for 
edge expansion start points. In the preferred implementation local gradient maxima which meet the magnitude condi- 
tion are selected as the edge expansion start points. A square window centered at the non-defect pixel adjacent to the 
extended defect region is searched for the local gradient maximum. Only non-defect pixels that are not points of any 
already detected edge are searched for the gradient maximum in the window. In the preferred implementation the size 
of the search window is equal to required minimum edge length LM IN. The search window is moved along the perimeter 
of the extended defect region in the loop represented by steps [401][460][462] In step [401] the search window ie posi- 
tioned at the first non-defect pixel adjacent to the extended defect region encountered when the image is scanned. The 
window is searched for the maximum gradient magnitude [402]. If the maximum is smaller than a specified magnitude 
threshold MM IN [404] then a new defect neighborhood is selected [462] if available [460]. If the magnitude is greater 
than the threshold MM IN the start edge point, 8, is -found and the edge expansion process ie initiated. 

If the gradient magnitude is greater than threshold MMINforthecun-ent edge point then the edge direction is deter- 
mined [406]. First, an average gradient directional angle Is calculated tor the square window centered at the current 
edge point. The average edge directional angle is perpendicular to the calculated average gradient directional angle. 
The size of the averaging window depends on the noise level of the image and the size of the used blur operator. For 
the wide range of noise levels and associated blur sizes the 3x3 averaging window gives good results. 

The average edge directional angle -for the current edge point is checked [403] if it is within the specified angle tol- 
erance AMAX of the average edge directional angle calculated for the previous edge point. This test is not performed 
tor the start edge point when there Is no previous edge point available. If the angle test falls then the maximum gradient 
search window is moved [462] to the next non-defect pixel adjacent to the extended defect region and the process starts 
again [402]. If the edge directional angle test is passed [406] and the maximum allowed edge length LMAX (equal -to 5 
times the maximum defect width) is not reached [41 0] and the expansion is in the forward direction [41 1 ] then the next 
closest pixel to the current edge point In the edge expansion direction Is selected [412] as the new current edge point. 
If the new edge point is not in the extended defect region [414] then the magnitude test is performed [404] and the proc- 
ess continues as described above. If the extended defect region is entered [414] then the current edge length is 
checked [416]. If the required minimum edge length LMIN is not reached [416] and the edga expansion direction is to 
intersect the defect region (the forward expansion direction) [416], then the current edge point is reset to the start point 
and the edge expansion direction is reversed [420]. 

The flow of control tor the reverse edge expansion process Is the same as described above tor the toward edge 
expansion with the exception that the reverse expansion stops when the minimum edge length LMIN is reached [411]. 
When the extended defect region has been entered and the defect entry part of the edge has at least the minimum 
number of required points LMIN then the statistics (average and variance) are calculated for the AO, A1 areas [422]. 
These statistics are later used to test a color match tor the connected edge. 

The defect entry direction is calculated [424] as the average edga directional angle of the specified number of edga 
points preceding the defect entry edge point. The number of the edge points used to calculate the defect entry direction 
is the minimum of the half defect width and the half length of the edge defect entry part in the preferred implementation. 
Alternatively, the edge properties e.g. the edga curvature and the edge directional angle variance can be used to deter- 
mine the number of points for calculating the defect entry direction. The edge expansion process is continued through 
the extended defect region in the defect entry direction until the defect exit point is reached [426][428][430]. The expan- 
sion process is aborted when the maximum allowed edge size LMAX is reached [426]. The edge expansion of the 
defect exit part of the edge is continued in a similar way as in the forward expansion of the defect entry part of the edge. 
For each edge point the gradient magnitude is checked [432] if greater than the specified threshold MMIN, and the edge 
directional angle is checked [436] if within a specified tolerance AMAX of the edge directional angle of the preceding 
edge point. However. In the case of the first point of the defect exit part (that Is, the point adjacent to the extended defect 
region) the edge directional angle is checked if within a specified tolerance AMAX of the defect entry angle. If one of 
these tests fails the expansion is stopped. Also the expansion is stopped when the defect region is reached [442] or 
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when the total edge length exceeds the maximum allowed length LMAX or the defect exit edge part length le equal to 
the defect entry edge part length [438]. 

When the defect exit edge part expansion ie stopped the exit edge part length ie checked [444] if greater than or 
equal to the previously specified required minimum edge length for the exit part. If not the edge ie stored in the edge 
map as a not connected edge [450] consisting of defect entry part and defect part, Otherwise the statistics of the A2 
and A3 areas are calculated [448] and ths color match Is tasted [448] be described previously. If the color match test is 
passed for both area pairs A0.A2 and Ai , A3 then the edge is stored [452] in the edge map as a connected edge con- 
sisting of defect entry part, defect part and defect exit part. Otherwise the edge is stored [450] in the edge map as a not 
connected edge. The edge expansion process is completed at that point. If the whole edge detection area in the image 
has baen searched for edges [480] (i.e. the entire non-defect region about and adjacent to the defect ragion) the edge 
detection step Is completed. Otherwise the edge detection process Is continued [462]. 

z. Edge Map Manipulation 

In the edge detection step the edge map has been created. Based on the edge map, the edge manipulation proc- 
ess is performed. The edge manipulation process consists of two separate steps shown in Fig. 5a and Fig. 5b. In the 
first step, the edge map is scanned for edges and a thinning operation is performed on edges as shown in Fig. 5a. When 
the first step is completed for all edges in the edge map the second step is performed. In the second step, the edge map 
processed in the first step is scanned for edges and a geometric extraction operation is performed on edges as shown 
in Fig. 5b. Referring to Fig. 5a, the first step of the edge manipulation process will be described. Edges from the edge 
map are scanned in the loop represented by steps [50£][517][518]. H is assumed there are some edges in the edge 
map. The first edge is selected from the edge map in step [502]. For the selected edge £/, the edge manipulation is per- 
formed in steps [503]-[5iB]. If the selected edge is the last edge in the edge map [517] then the first step of the edge 
manipulation process is completed. Otherwise the next edge is selected from the edge map [5 16] and the process con- 
tinues. For the selected edge B { , the edge manipulation starts with initializing empty edge list LB with the edge B f [503]. 
Next, edges from the local neighborhood of the detect entry part of the edge B, are added to the list LB [504]. In the 
preferred implementation in step [504], all edges that intersect a lina segment centered at the selected point of the 
defect entry pari of the edge 8 ( are added to the list LB. The line segment direction is perpendicular to the edge Sy at 
the selected edge point. The selected edge point is located a specified number NP of edge points from the defect entry 
point of the edge B t between the defect entry point and the first point of ths edge B h The number NP of odge points is 
equal to haH of thB required minimum edge length LMIN. Tha fine segment length is equal to 2g + 2b pixels where 2g 
+ 1 1s the gradient operator size and 2b + 1 1s the blur operator size. 

If the edge B i is connected [506] then the edges from the local neighborhood of the defect exit part of the edge B { 
are added to the list LB [506]. In the preferred embodiment, the edge search in the local neighborhood of the defect exit 
part of the edge B f [50$] is implemented in an analogous way as for the defect entry part [504]. 

When the list LB Is complete all edges on the list thai do not match with the edge & t are removed from the list LB 
[510]. Two edges on tha list LB match if their defect entry angles are within a specified angle tolerance. In the preferred 
implementation, the angle tolerance for the edge match is equal to 15 degrees. The best edge BB { is selected from the 
remaining matched edges on the list LB [51 1]. H there are any connected edges on tha list, the strongest of the con- 
nected edges is selected as the beet edge B8 { . Otherwise the strongest of the not connected edges on the list is 
selected as the best edge BB r The edga strength is equal to the average gradient magnitude of its edge points. 

If the best edge BB } Is not connected [512] the connecting operation Is performed for the edga BB { In steps [513]- 
[515]. In the connecting operation two not connected edges can be detected as the line geometric primitive and 
replaced in the edge map by the line primitive. In the preferred implementation the list LG of edges from the local neigh- 
borhood of the defect exit point of the not connected edge BBj is created [51 3] using a square window centered at the 
defect exit point. The size of the window Is equal to 2THC + 1 . The threshold THC Is defined below In a line geometric 
primitive specification. Next, the list LC is scanned and -tor each not connected edge from the list LC and the not con- 
nected edge BB { the geometric and color match constraints of a line geometric primitive are checked [51 4] . These con- 
straints are listed below in a line geometric primitive specification [Fig. 6a). A not connected edge from the list LC 
detected as a line geometric primitive with the best color match gets connected with the edge BB f and a line geometric 
primitive extraction is performed [515]. The edge B8 ( and the selected not connected edge are replaced in ths edge 
map by the new connected edge BB } representing a line primitive. This operations is described below in a line geomet- 
ric primitive specification (Fig. 6a) . 

Next, all but the best edge BB t on the list LB are removed from the edge map [516]. Tha edga thinning operation 
for the edge S; is completed at that point. If the edge B, is tha last edge ths first step of the edge manipulation process 
is completed. Otherwise the next edge is selected [518] and the process continues [503]. 

Whan the first step is completed the second step of the edge manipulation process Is performed (Fig. 5b). The 
edge map processed in the first step is scanned in the second step for the edges in the loop represented by steps 
[524][59Q][592j. It is assumed there are some edges in the edge map. The first edge is selected from the edge map in 
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step [524]. For the selected edge BB it the geometric primitive extraction operation Is performed In steps [526]-[582]. If 
the selected edge is the last edge in the edge map [590] then the second step of the edge manipulation process is com- 
pleted. Otherwise the next edge is selected from the edge map [592] and the process continues. 

The selected edge BB f is followed in the edge map from its first point to its last point and a geometric primitive is 
s extracted based on the Intersected edges. The geometric primitives supported In the preferred Implementation, 1he cor- 
responding conslrEinrts and extraction operations are described in the next section (Fig. 8a-Bf). A geometric primitive is 
detected if all of its geometric and color match constraints are satisfied. The selected edge must be a part of the 
detected geometric primitive. Different geometric extraction methods ere used for the connected and not connected 
edge BB f [525]. 

io If the selected edge BB { is connected [526] the points of the edge B0, are selected in the loop represented by steps 
[528][546][546]. The edge point selection loop starts with the first point of the edge B8 f [528]. The geometric primitive 
extraction operation is performed in steps [530]-[542]. If the selected edge point Py is not the last point of the edge BB; 
[545] then the next edge point is selected [548] and the process continues until the end of the edge is reached [546]. At 
the selected edge point P j the edge map is checked for the intersected edges [530]. If there ere no intersected edges 

is [530] the next point of the edge BB f is selected [546][548]. Otherwise a check is performed If the intersected edge is 
connected [532]. Depending on whether the intersected edge is connected or not connected [532] different geometric 
and color match constraints of supported geometric primitives are checked [540][534], The detection of geometric prim- 
itives in steps [540][5S4] depends also on the intersected edges found at the preceding points of the edge BB;. Next, 
the geometric primitive extraction is performed [536][542] based on the type of the detected geometric primitive. For the 

20 connected intersected edge the X-junction primitive, when detected, is extracted in step [536]. For the not connected 
intersected edge the T-junction primitive, when detected, or the background feature primitive, when detected, is 
extracted in step [542]. If the not connected intersected edge is not a part of any geometric primitive then it is removed 
from the edge map [542]. When the end of the edge BB f is reached [546] the final edge cleaning operation is performed 
[548]. If the connected edge BB; is intersected by some connected edges and no geometric primitive is detected then 

as the edge BB { - and all its intersected edges are removed from the edge map [548]. At that point the geometric primitive 
extraction operation Is completed for the connected edge BB}. 

If the selected edge BBf is not connected [560] a different geometric primitive extraction method is used. The edge 
B8 ; . is followed starting with its first point [560] in the loop represented by the steps [564][565]. At the selected edge 
point Pj the edge map is checked for the intersected edges [562]. If there are no intersected edges [552] the next point 

30 of the edge BBf is selected [564][566]. Otherwise a check is performed if the intersected edge is connected [563]. If the 
intersected edge is not connected [588] the geometric and color match constraints of the corner primitive are checked 
[580]. If the corner Is detected then the corner extraction operation Is performed [582] and the edge manipulation for the 
selected edge 8 { is completed [590]. If the corner primitive [560] is not detected then the intersected edge and the edge 
B8{ are removed from the edge map [584] and the edge manipulation for the selected edge BB t is completed [590]. If 

35 the intersected edge is connected [568] then a part of the edge BB; extending from the point P { to the last point of BB; 
Is removed from the edge map [570] and the edge manipulation for the selected edge BB S is completed [590], If the not 
connected edge SB, does not intersect any edges [584] it is left in the edge map and the edga manipulation for ths 
selected edge B8j is completed [590]. 

As a result of the edge manipulation step only extracted geometric primitives and single not connected edges are 

40 stored in the edge map. 

Fig. 8a-6f shows all geometric primitives supported in the preferred implementation: a line primitive (Fig. 3a), a cor- 
ner primitive (Fig. 8b), a background feature primitive (Fig 6c), an open T-Junctlon primitive (Fig. 6d), a closed T-Junctlon 
primitive (Fig. 6e), an X-junction primitive (Fig. Sf). 

Each geometric primitive is defined by geometric constraints and color match constraints. The geometric con- 

45 straints define a unique edge geometric configuration. The color match constraints define pairs of edge adjacent areas 
that must match In color with a given confidence level. The color matching Is Implemented the same way as for the 
detection of the connected edges described in the previous section. The edge adjacent areas for the color match test 
extend a given distance perpendicular to the edge in the non<tefect region. In the preferred implementation the area 
extension distance is equal to fie half defect width. The mean and variance statistics for each color channel of an image 

so are calculated for these areas. There is a color match between two areas if the Student's t-test is passed for each color 
channel statistics with a specified confidence level. En the preferred implementation 0.3 to 0.6 confidence level has 
given good results for a wide range of images, 

Ling Cggmgtric Primitive 

55 

A line geometric primitive is described below. Fig. 6a shows a line geometric primitive. Each connected line is a line 
geometric primitive. A connected line does not require any extraction operation. IVvo not connected edges E0.E1 are 
detected as a line geometric primitive if the following geometric and color match constraints are satisfied. 
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Geometric constraints: 

Two not connected edges E0.E1 must enter the defect region -from the opposite sides. The defect entry angles of 
two not connected edges E0.E1 must be within a specified angle tolerance. In the preferred implementation 15 degrees 
5 is used as the angle tolerance for the angle difference normalized to 0-90 degress. The minimum of distances bstwesn 
defect exit points and defect entry pointsof two not connected edges E0.E1 mustbe not greater than a specif ied thresh- 
old THC In the preferred implementation THC Is equal to g + b + 0.26!.^ where 2g + 1 is the gradient operator size, 
2b + 1 is blur operator size, L^ er is the maximum length of the defect part of edges E0,E1 . 

w Color match constraints: 

There must be a color match for the following pairs of areas (A1,A2), (AO, A3). 

Extraction operation: 

is 

Two not connected edges E0.E1 that are detected as a line geometric primitive are replaced in the edge map by 
the edge E3. The edge E3 is a line segment through the point P which is the midpoint of the line segment connecting 
defect entry points Off edges E0.E1 . The direction of the edge E3 is defined by the average of two defect entry angles 
of edges E0.E1. The edge E3 extends on both sides of the defect region by the number of non-defect points corre- 
al spending 1o the lengths of defect entry parte of edges E0,E1 . 

Corner Geometric Primitive 

A corner geometric primitive is described below. Fig. 6b shows a corner geometric primitive. 

Geometric constraints: 

A corner geometric primitive consists of two intersecting parts of the not connected edges E0,E1 . The corner prim- 
itive edge parts extend from the edge first points to the intersection point of ED and El. The intersection angle of edges 
20 E0.E1 must be within a specified angle range. In 1ha preferred Implementation the 65 to 115 degrees range is used. 

Color match constraints: 

There must be a color match for the following pairs of areas (A1,A2), (AO, A3). 

35 

Extraction operation: 

The edge parts extending beyond the intersection point to the edge last points are removed from the edge map. 
40 Background Feature Geometric Primitive 

A background feature geometric primitive Is described below. Fig. 6c shows a background feature primitive. 
Geometric constraints: 

46 

A background feature geometric primitive consists of the connected edge E0 and a part of tha not connected edge 
E1 that extends from its first point to the intersection point with the edge ED. The intersection angle of edges EO, E1 
must be within a specified angle range. In the preferred implementation the 65 to 1 15 degrees range is used 

60 Color match constraints: 

There must be a color match tar the following pairs of areas (A1 ,AB),(Aa. A4) ,(A0, A5) . 

Extraction operation: 

55 

The pert of the edge E1 that extends beyond the intersection point to the last edge point is removed from the edge 
map. 
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T-Jundlon Geometric Primitive 

A T-junction geometric primitive is described below. Fig. 6d shows an open T-junction primitive. Fig. 6e showe a 
closed T-junction primitive. 

Geometric constraints: 

A T-junction geometric primitive consists of the connected edge EO and parte of the not connected edges E1 ,E2 
that extend from their first points to the intersection points with the edge EO. The intersection angles of E0.E1 and E0.E2 
must ba within a specified angle range. In the preferred implementation the 65 to 1 15 degrees range has been used. 
Additional geometric constraints can Include a minimal distance between the edges E1.E2 and/or a match of defect 
entry angles of ihe edges E 1 ,E2 within given angle tolerance. 

Color match constraints: 

There must be a color match for the following pairs of areas: (A1,A2),(A3 t A4) ( (A5 l A6) l (A0 l A7). In addition for an 
open T-junction primitive a color match is required for (A4.A7) and (AO, A3). 

Extraction operation: 

The edge parts of E1 ,E2 that extend beyond the intersection point with EO to the last edge pointsare removed from 
the edge map. In addition for an open T-junction primitive the part of the connected edge EO between the intersection 
points with the edges El ,E2 is removed Irom the edge map. 

X^uncjipn Geometric Primitive 

An X-junction geometric primitive is described below. Fig. 6f shows an X-j unction geometric primitive. 

Geometric constraints: 

An X-junction geometric primitive consists of four connected edges E0,E1,E2,E3. The edges EO, E1 intersect tha 
edges E2.E3, The Intersection angles must be within a specified angle range. In the preferred Implementation the 65 to 
115 degrees range has been used. Additional geometric constraints for a X-junction primitive can include a minimal dis- 
tance between edges E0.E1 and E2.E3 and/or match of defect entry angles of edges E0.E1 and E2 P E3 within a given 
angle tolerance. 

Color match constraints: 

There must ba a color match for the following pairs of areas: 
(Al^.tAS.A^.tAS.ASJ^A^ASJfAg.AIOXtAII.A^J^AIS, A14],(A0 f A15). Because all edges of the X-junciion primi- 
tive are connected there is a color match for the following pairs of areas: 
(A5 I A14) I (A6,A13),(A1 I A10) I (A2 I A9), 

Extraction operation: 

The edge parts between Intersection points are removed from the edge map. 

3. Correction 

In the correction step, values for each defect pixel tor each color channel of an image are provided based on the 
local non-dafect pixels. A local gdge map is used to determine a correction method. Two correction methods are inple- 
mented in the preferred embodiment. If there are any local geometric primitives, a given defect pixel is corrected using 
weighted averaging of predicted values from the local geometric primitives. Otherwise a modified pinwheel correction 
method is used. 

The correction method based on the local geometric primitives combines in a weighted -fashion estimated values 
from 1-D predictors. Two types of 1-D predictors are used In the preferred implementation: an extrapolation predictor 
and an Interpolation predictor. A 1-D extrapolation predictor uses non-defect pixels on one side of a delect pixel along 
a line segment in a given direction through the defect pixel. Any standard 1-D extrapolation predictor can be used e.g. 
nearest neighbor, least-squares linear fit, or the like. In the preferred implementation the least-squares linear fit is used. 
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A 1 -D Interpolation predictor uses non-defect pixels on both sides of a defect pixel along a line segment In a given direc- 
tion through a defect pixel. Any standard 1-D interpolation predictor can be used, tor example, nearest neighbor, least- 
squares linear fit, or the like. In the preferred implementation the least-squares linear fit is used. Directions of the line 
segments for the extrapolation and interpolation predictors are determined by the local geometric primitives. 

s Fig, 8a-8f shows the line segment directions for extrapolation and Interpolation predictors for each geometric prim- 
itive. In Figure Sa-Sf edges or parts of edges that are parts of a detected geometric primitive are shown for simplicity as 
line segments although as described in the edge detection step they may have non-linear shape in the defect entry or 
defect exit parts. Dashed straight lines with arrows on both ends represent line segment directions for the interpolation 
predictors. Dashed straight lines with an arrow only on one end represent line segment directions tor the extrapolation 

to predictors. The line segments used by extrapolation or interpolation predictors at a given defect pixel are in directions 
corresponding to directions of the neighboring parte of the local geometric primitives and must not Intersect any geo- 
metric primitive. A direction of a geometric primitive part is defined by the defect entry angle of the corresponding edge. 
A "neighboring'' part of the geometric primitive for a given defect pixel must not be separated from the defect pixel by 
any other part of a geometric primitive in the sense that the shortest line segment connecting the defect pixel and the 

'£ neighboring pert does not intersect any geometric primitive. For example, for the defect pixel PO in Fig. 3d only EO and 
E1 are neighboring parts. If a defect pixel lies on a part of a geometric primitive then the line segment extends along 
that part of the geometric primitive and it must not intersect other parts of a geometric primitive. 

A line segment tor the extrapolation predictor extends only on one side of the defect pixel and it must include a 
specified number of non-defect pixels. The number of non-defect pixels depends on the used extrapolation predictor. A 

a* line segment for the interpolation predictor extends on both sides of the defect pixel and it muet include a specified 
number of non-defect pixels on each side The number of non<defect pixels depends on the used interpolation predictor. 
For the nearest neighbor interpolation/extrapolation predictor only one non-defect pixel is required on one side of the 
corresponding line segment. In the preferred implementation the least-squares linear fit extrapolation and interpolation 
predictors are used and the required number of non-defect pixels of the line segment on one side of the defect pixel is 

as equal to haK of the defect width. If for a given defect pixel a line segment meeting the foregoing requirements extends 
on both sides of the defect pixel an Interpolation predictor Is used, otherwise If the line segment extends only on one 
side of the defect pixel an extrapolation predictor is used. 

There is a one to one correspondence between each predictor and neighboring part of the local geometric primi- 
tives. The total number of predictors is equal to the total number of neighboring parts of local geometric primitives tor a 

so given defect pixel . Fig. 6c shows example line segments for interpolation and extrapolation predictors. Two extrapolation 
predictors (line segments LeO.Lel) are used tor the defect pixel PO. One interpolation predictor (line segment LiO) is 
used tor the defect pixel PI , 

In the correction step the final values of the defect pixel are the weighted sum of all estimated values irom extrap- 
olation and interpolation predictors tor each color channel, where the weights are inversely proportional to the distances 
ss from the defect pixel to the neighboring parts of the local geometric primitives. The distance between a defect pixel and 
a neighboring part Is the length of the shortest line segment connecting the defect pixel and the neighboring part. The 
distance is clipped to tha minimum limit greater than 0. The final values tor the defect pixBl are given by: 

* Pw-E^tfJr (EQ.1) 

>1 



where: 

46 

Pijk - value tor the ftth color channel of the Ah defect pixel In the source Image 

W S j - weight of the jth predictor for the flh defect pixel in the source image 

so 
ss 

where: D u Is the distance of the /th defect pixel from the neighboring part of the geometric primitive tor the flh 
predictor; and C(d) is the clipping function defined in the following way 
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tf 3, otherwise 



d } d)dl 



s 



in the preferred implementation d?=1 and rt?=0.5; 



10 



E { j ik - estimated value for tha fcth channel of the rth defect pixel from the /th predictor ; and 



J: - number of predictors for the /th defect pixel 

Fig. 7 shows a flow chart of the correction process. The defect pixels are scanned in the loop represented by steps 

75 [702][792][794]. It is assumed there are some defect pixels in the image [702]. For each defect pixel local geometric 
primitives are detected [704-71 4]. Next, if any local geometric primitives are detected a correction method based on the 
local geometric primitives [716-742] is performed. Otherwise a modified pinwheel correction method [760-784] is used 
to provide new defect pixel values. 

Assume P f is a selected rth defect pixel. First, a local neighborhood of the selected defect pixel is checked for the 

?o local geometric primitives in the edge map. A set of N line segments through the defect pixel ie created in the edge map 
[704]. The line segments are at equal radial angle intervals with respect to each other about the defect pixel. The 
number of line segments depends on the size of the angle interval. In the preferred implementation 22.5 degrees is 
used which corresponds to 8 line segments Preferably, each line segment is composed of pixels which are less than 
0.5 pixel distance away from a line through the defect pixel at the radial angel associated with each respective line seg- 

& merit. The set of line segments defined above is referred to as a pinwheel. Each line segment is extended in the edge 
map on each side of the delect pixel starling from the defect pixel until one of the following tour conditions are met: (I) 
a part of a detected geometric primitive ie intersected; (ii) a specified number of non-defect pixels on each side of tha 
defect pixel is reached; (iii) a maximum line segment length is reached; (iv) or an image boundary is reached. In the 
preferred implementation the maximum number of non-defect pixels on one side of the defect pixel along the line seg- 

30 ment is equal to half the defect width and the maximum line segment length equals five times the defect width. 

Each part of a delected geometric primitive NP;j intersected by the pinwheel line segments and the corresponding 
distance between the Intersection point and the defect pixel are stored on a list. Each Intersected part Is a neighboring 
part due to the fact it is the first one in a given pinwheel direction from the defect pixel When all line segments are 
extended the list of neighboring parts is processed [704] to remove duplicated geometric primitive parts on the list and 

35 to find the minimum distance for each unique part WPy on the list. The result is the list of all neighboring parts NP S j 

of the local detected geometric primitives for the selected pixel P ft where j = 1 JJ. If no geometric primitive Is detected 

[70S] that means the list is empty then the modified pinwheel correction method is used [760-784]. Otherwise the 
minimum distance dmin on the list is found [710]. dmin is the distance between the selected pixel P $ and the closest 
neighboring part. Next, a test is performed to determine if there is an open direction at tha selected pixel P ; [712]. Thera 

40 is an open direction if there is at least one pinwheel line segment with a specified number of non-defect pixels on each 
sideof the defect pixel and with no interactions with any geometric primitive. In the preferred implementation a required 
number of non-defect pixels on each side of the defect pixel along a line segment In tha open direction Is equal to half 
of the defect width. If there is no open direction the correction method based on the local detected geometric primitives 
[71$] starts. If there is an open direction the minimum distance dmin is checked against a distance threshold dl [714]. 

4s in the preferred implementation the distance threshold d1 equal to half of the defect width is used. If the minimum dis- 
tance dmin Is greater than the distance threshold d1 then a modified pinwheel correction method [760] starts. This 
method is described in the next section. Otherwise the correction method based on the local detected geometric prim- 
itives [716] starts. 

For the correction based on local detected geometric primitives, the sum of estimated pixel values P itk for the rth 

so defect pixel is initialized lo 0[716] for each color channel of an image k = 1 K where K is the number of the color 

channels. Also the sum of weighting factors W } is initialized to 0[716] The list of neighboring parts NP fJ of the local 

detected geometric primitives for the selected pixel F/. where j = 1 J fl is scanned in the loop represented by steps 

[718][734][736], For a given neighboring part NP f j a corresponding interpolation or extrapolation predictor is deter- 
mined [720]. The predictors are described above and shown in Figures 3a-6f. If there is no valid predictor [722] due to 
55 the defined constraints the next neighboring part is selected [734][73S] and the processing loop continues [720] . If there 

is a valid predictor [722] the values of the /th defect pixel £/j,*are estimated for each color channel k = 1 /Cueing >th 

predictor corresponding to the/th neighboring part, The clipping of the neighboring part distance D fJ Is Implemented In 
steps [726J72B]. In the preferred implementation d2 = 1 and d3 = 0.5. Based on the clipped distance D f j weighted esti- 
mated pixel values are added to tha sum P ifk [730] for each color channel ic= 1 K. A weighting factor inversely pro- 
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portlonal to D }j Is added to the sum W f [730]. The processing loop continues [736][720] until the lafit neighboring part 
is processed [734]. If no pixel values are estimated [740] than the modified pinwheel method starts [760] otherwise the 
sum of weighted estimated values P ijk is normalized by the sum W, of weighting factors [742] for each color channel 
= 1 K. The final values of P^ are defined by the equation (EQ. 1). The calculated detect pixel values are stored in 

5 the corrected Image [790]. 

If the correction method based on the local geometric primitives cannot be used [708-714] or it fails [740] the mod- 
ified pinwheel method [760-754] is used instead. The pinwheel correction method is described in detail below, tt is 
referred to as a standard pinwheel method. In the preferred implementation the pinwhael method starts with creating N 
pinwheel line segments throuQh the defect pixel P ( [760] in the source image. Any of the line segment creating/expand- 

w ing methods described in connection with the standard pinwhael method below can be used. All pinwheel line seg- 
ments are scanned In the loop represented by steps [762][774]|776], For each valid line segment Its figure of merit Is 
calculated [770]. A pinwheel line segment is valid [764] if two conditions are satisfied. First, there is & specified number 
of non-defect pixels on each side of the defect pixel along the line segment. This oonditicn is the same as in the stand- 
ard pinwheel method. Secondly, a valid line segment cannot intersect any geometric primitive. This condition is added 

is in the preferred embodiment compared to the standard pinwheel method, "me pinwheel created in the image [760] is 
duplicated in the edge map to check for the intersections with geometric primitives and to create the list LNC of local 
not connected edges [760] in a similar way as in step [704]. If a given pinwheel line segment is valid [764] then its total 
figure of merit F //p is calculated [770]. In the modified pinwheel method the total figure of merit for /rth pinwheel line seg- 
ment at rth defect pixel in the source image is calculated according to the equation EQ.2. The total figure of merit is 

so equal to sum of two components. One component is a figure of merit F v based on the uniformity of non-defect pixel val- 
ues along rfth pinwheel line segment measured by the total mean square error SMSB^ for nth pinwheel line segment 
at hh defect pixel in the source image. SMSE^ is calculated according to the equation EQ. 3. The more uniform non- 
defect pixel values along a given pinwheel line segment the higher figure of merit F v . The detailed description of the 
total mean square error for a pinwheel line segment can be found in the description below relating to the standard pin- 

£5 wheel method. A least-squares line fitting is used in the preferred implementation to calculate a functional model of the 
non-defect pixel values along a pinwheel line segment. The figure of merit F v Is calculated In the preferred Implemen- 
tation according to the equation EQ.4. The second component of the total figure of merit (EQ.2) is based on the local 
not connected edges. The list LNC of local not connected edges is built when the pinwheel is created in the edge map 
[750]. In the preferred implementation an alignment figure of merit with local not connected edges F A is calculated 

30 according to the equation EQ.5. F A has higher values for the pinwheel line segments aligned with directions of the local 
not connected edges. F A is scaled by the 'decay in distance' function DQ which decreases with increased distance of 
the defect pixel from the local not connected edge, The 'decay In distance' function DQ Is calculated according to the 
equation EQ.6 in the preferred implementation. 

When all pinwheel line segments are scanned [774] and there are no valid line segments [778] a weighted averag- 
es ing of non-defect pixels of the pinwheel line segments is used to calculate defect pixel values [784]. This method [784] 
Is described below In connection with the standard pinwheel method, If there are any valid line segments [774] the M 
valid line segments with the highest figures of merit are selected [780]. For each selected line segment at the rth defect 

pixel estimated pixel values for each color channel k=l f ,K are calculated [782] using an interpolation in each 

selected line segment direction! A weighted averaging of the estimated values is used to calculate final values of the 

40 defect pixel [782] according to the equation EQ.7 Weighting is proportional to figures of merit according to the equation 
EQ.8. The final defect pixel values are stored in the corrected image [790]. 

F,„ = F v (SMSE in )+k A ^{D^F^A^A^) (EQ.2) 

6 /.i 



Ff „ - total figure of merit for nth pinwheel line segment at /th defect pixel in source image 

FUSMSE Kfl ) - figure of merit for /rth pinwheel line segment at /th defect pixel in source image based on total mean 

so square error 

SMSE )n ■ total mean square error for nth pinwheel line segment at rth defect pixel in source image 

where 

55 K 

SMSE^-^MSE^ (EQ.3) 
fr-i 



10 
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K - number of source Image color channels 

JWSF; „ k - mean square error between norniefect pixel values and their lunctional model for Wh color channel ol 
nth pinwheel line segment at flh defect pixel in source image 

F A {AsjA n ) - alignment figure of merit of r?th pinwheel line segment at rth defect pixel In source image with /th local 
not connected edge 

Afj - defect entry angel of /th iocal not connected edge for Ah defect pixel in source image 

A n - angle of nth pinwheel line segment 

D[D f Ji - 'decay in distance' function 
is Dfj - distance between Ah defect pixel and its /th local not connected edge 

i f - nurrter of local not connected edges for rth defect pixel in source image 

k A - scaling factor equal to 1 in the preferred implementation 

In the preferred implementation figures of merit are defined in the following way: 

/ SMSEww-SMSEf* \ 



30 



25 



MAXQ ■ is a maximum function 



30 



SMSEmax. - 

35 maximum total mean equve error for all valid pinwheel line segments at rth detect pixel In source image 



SMSEMINi ~ 

40 

minimum total mean square error for all valid pinwheel line segments at Ah defect pixel In source Image 



45 

so \A f j-A n \ - absolute value of angle difference modulo 90 degrees 

Awuc ■ maximum angle difference threshold equal to 90 degrees in the preferred Implementation 
In the preferred implementation a decay in distance' function is defined in the following way: 

55 




(EQ.6) 
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Si - decay constant which, In the preferred Implementation, Is equal to minimum delect width at * defect pixel In 
source image 

The final corrected pixel values are calculated according to the following equation: 

'tf-2>u*i* (EQ7) 



'A' 



final corrected value of /th defect pixel In source Image for fcth color channel 
M/ y n - weight of nth pinwheel line segment at rth defect pixel in source image where 
75 s set of indices corresponding to selected pinwheel line segments 

W Kfl - -^2- (EQ.8) 

£y,n,* ■ interpolated value of / th defect pixel in source image for fcth color channel along nth pinwheel line segment 
25 Standard Pinwheel Method 

The standard pinwheel method described below, while computational!/ efficient and yielding good results in many 
cases, may not provide an ideal result when filling long regions that intersect objects where the object width is smaller 
than or comparable to the width of the region. This could lead to the appearance of object separation and boundary dis- 
20 tortions in the restored image. The operation of the standard pinwheel method, is described in detail below with refer- 
ence to Figures 2(a-c] and 9-14. 

When computer system 10 Is started at step 1098, system 10 receives a digital Image from Image reader 20 and 
stores the image in memory 14 (step 1000). Alternatively, this image may have already been stored in memory 14. The 
image stored in memory 1 4 will be referred to as the ' source image." An example of a source image 25 is shown in Fig- 
35 ure 2(a) with defect region 26. The source image may be comprised of a single color channel (monochrome), or have 
multiple color channels, such as RGB. 

Once the source image is stored in memory 14, than in step 1102 the defect regions in the source image are 
located. This maybe done in 1he same manner as described above in connection with the method of the present inven- 
tion. 

40 Once -the defect regions are located, a defect map is built and stored in memory 1 4 (step 1 1 04). In the defect map, 
pixels with a specified valua correspond to defect pixels in the source image. One example of a defect map is shown in 
Figure 2(b). 

In the standard pinwheel method, a window operator is set, preferably centered, on a -first pixel in the source image 
(step 11 05). In computer system 10, the window operator is used in performing correction of defect pixel values in the 

*b source image. The window operator defines a region of interest in the source image about a pixel centered within -the 
region. In one embodiment the shape and size of this window operator Is a square region, where each side or boundary 
of the square region is X in size. However in another embodiment, the window operator may be a circular region having 
a diameter X. This window operator is used by processor 12 to scan across each pixel in the lines of the source image. 
The scanning pattern of the window operator is not critical as long as all defect pixels are visited during scanning. For 

so example, a raster-scan pattern may be used. 

In step 1 107, a check is made to determine whether this pixel set within the window operator is a defect pixel by 
referencing the defect map stored in memory 14. If the pixel Is a non-defect pixel, then In step 1 109 the values of this 
non-defect pixel are stored in an area of memory 14 which is allocated for storing the corrected source image f 'cor- 
rected imagB"). If the pixel ie a defect pixel, then the defect pixel is selected (TIXELsel") tar subeequent correction 

35 (step 1 106). Next, a corrected value in each channel of P IXEL SEL is estimated (step 1110). The process for estimating 
corrected values is described in greater detail in Figure 10. 

Referring to Figure 10, the process tor estimating corrected values for PIXEUsel Is started at step 1 198. The line 
segment size is set to X for every direction (step 1 200). Preferably, X is at least twice the maximum width of the defect 
region in the source image. A SET of line segments or vectors is defined or allocated through PIXELsel wherein each 
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line segment Is composed o! both defect and nornd&ted pixels In the source Image on one or both sides of PIXEL S a_ 
(step 1Z02). In this embodiment, the total number of pixels defect and non-defect pixels, in each line seQment is main- 
tained the same, which corresponds to the square window operator. Alternatively, aach line segment can have the same 
length X defined as the Euclidean distance, which corresponds to the circular window operator, However, in other 

5 embodiments the total number of pixels In each line segment may vary depending on the defect pixel and line segment, 
as will be described later. 

SET is composed of N number of line segments, where each line segment is represented by the fallowing notation: 
In SET n refers to each line segment (n-1 to N). Each line segment n has pixels having values for K number of color 
channels. The nth line segment is referred to as LS n . The fth channel of the nth line segment in SET is referred to as 
10 LS n k . For example, where SET has 4 line segments (N-4), and each pixel has 2 channel values (K-3), the 2nd channel 
of the 3rd line segment In SET Is LS^ 2 . 

Preferably, the line segments are each at equal radial angles at 45° or 22.5° intervals with respect to each other 
about P IXELs EL , although the angles can vary as needed. The number of line segments in SET depends on size of this 
angle. For example, radial angles at 45° intervals provide 4 line segments in SET, while radial angles at 22.5* intervals 
is provides 8 line segments in SET. Preferably, each line segment is composed of pixels which are less than 0 5 pixel dis- 
tance away from a line through PIXELa EL at the radial angle associated with each respective line segment. 

Referring to Figure 14, an example of a SET of 4 line segments (H, V. Dl , and D2) is shown at 45° spacing to each 
other, where each line segment is composed of a total of 22 pixels and has 1 1 pixels on each side of P IXEL SEL . 

Referring back to Figure 10, once the SET of line segments is defined, then at step 1204 any line segments which 
so do not contain a minimum threshold number of non-defect pixels on each side of PIXEL SEL are considered not valid line 
segments, and are removed from SET. Preferably, the minimum threshold is set to half the number of pixels on one side 
of a line segment. For example, if each line segment has 8 pixels, then 4 pixels are on each side of PIXEL SEL and tha 
minimum threshold is 2 pixels. Accordingly, any line segment which does not have at least 2 non-defect pixels on each 
side of PIXELS is removed from SET 

6 Next, a check is then performed to determine whether the remaining number of line segments N in SET is greater 
than zero (step 1206). H N Is not greater than zero, then the W branch Is taken to step 1207 where all the line seg- 
ments previously removed are replaced in SET Once the line segments are replaced, a query is made to determine 
whether any of the line segments have non-defect pixels (step 1209). If there are no non-defect pixels in the line seg- 
ments of SEX then the "no" branch is taken to step 121 1 where the line segment size X is increased by Y This accord- 

30 ingly expands the window operator region by Y, thereby increasing the possibility that when SET is redefined line 
segments will include non-defect pixels. A new SET of line segments is redefined through PIXEL SEL at this increased 
line segment size at step 1202, and the process as described above Is repeated. If the line segments at step 1209 con- 
tain any non-defect pixels, then the "y*&" branch is taken to step 1213 where an estimated value for each channel k of 
PIXELsel is calculated based on the weighted average of the non-defect pixels in SET. These weights are determined 

35 by their respective distances Ircm PIXEL sa _, 

The calculation for estimating values tor each channel k of PIXEL SEL at step 1213 preferably Is as follows: 



* POEL^-fclg (EQ.9) 

/■1 



45 where: 



I ■ index for aach non-defect pixel in SET; 

Q = total number of non-defect pixels in SET; 

(\k - the value of the Ath channel of the Ah non-defect pixel in SET; and 

so W| = the weight tor the /th non-defect pixel in SET, where 



55 

x, and Yi = the x and y coordinates of the i* 1 non-defect pixel in SET in the source image, and 
Xibel and y SE L = the x and y coordinates of PIXELs EL In the source Image 

Once the estimation of values for PIXELs EL is completed (step 1218) processing then goes to step 1 1 12 in Figure 
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9. 

If at step 1206 the number of line segments N is greater than zero, then the "yes" branch is taken to step 1208 
where a functional model is calculated "for the non-defect pixel values "for each channel k of each line segment in SET. 
The functional model calculation is performed by -fitting 1he values of the non-defect pixels in a line segment to a math- 
$ ematlcal model, such as a flat field fit, linear fit, or quadratic fit. This fitting Is based on the values of the non-defect pix- 
els, and the distances between tham. A variety of techniques may be used for modeling the non-dafect pixel values, and 
the particular technique chosen will affect model accuracy, and require a different minimum number of non-defect pix- 
els. One example of this technique is shewn in W. PRESS, S. TEUKDLSKY W. VETTERLING, AND B. FLANNERY, 
NUMERICAL RECIPES IN FORTRAN, Ch. 15 (2d ed, 1992). In the preferred embodiment, the functional model Cai- 
ro dilation at step 1208 uses a linear least-squares fit, which requires at least one non-defect pixel lying on each sfcfe of 
PIXELsel Earlier performed step 1 204 may be used to assure that this minimum Is met for all line segments En SET. 

Next at step 1210, a mean square fitting error (MSE) is calculated of the non-defect pixel values from the functional 
model for each channel of each line segment in SET. The MSE calculation is a statistical measure off the deviation of 
the non-defect pixel values in a line segment from an estimate of their values based on their functional fit. Thus, the 
15 lower a M5E value the mere consistent to the model are the non-defect pixel values. For a given channel and line seg- 
ment, MSE is calculated by determining the difference between the value of each non-defect pixel from its value esti- 
mated along the functional model. The resulting difference for each non-defect pixel is then squared, and the sunn of 
these squares is averaged lo obtain the MSE. (In the case of aflat field fit for the functional model calculation, the MSE 
is equivalent to the variance of the non-defect pixels in the line segment.) Mean square fitting error of the Ath channel 
20 of the :7th line segment (MSE n k ) is calculated by: 

n /-I 

where: 

M n = number of non-defect pixels in LS n ; 
30 I ■ index for each non<lsfect pixsl in LS n from 1 to M n ; 

p i,n,k ■ 1he actual value of the ith non-dafect pixel in channel k in LS n ; and 

Pi,'n,'k = the functional model estimate value of the /th non-defect pixel In channel k In L5 n . 

Then at step 1212, a calculation is performed for each line segment wherein each channel of the line segment is 
35 combined to provide a total mean square fitting error value 

<MSE Totaln ), 



40 where: 



MSE^ml^MSE^ (EQ.12) 

45 

K ■ the total number of channels (tor ROB, K-3); and 
k = index for each channel from 1 to K. 

so The line segment having the lowest MSE To1C) | is then selected (step 1214). The selected line segment represents 
the preferred line segment for determining estimated values far PIXEL SEL . An estimated value tor each channel k of 
PIXELsel is then interpolated using at least one non-defect pixel of the selected line segment on each side of P IXEL^ EL 
(step 1 21 6). In this manner, by selecting the line segment whh the lowest total mean scare fitting error, the line segment 
representing non-defect pixels which are most consistent to their functional model across all channels is used in deter- 

ss mining the estimated values tor PIXEL SEL . Once the estimation of values tor PIXELsel is corrpleted (step 1218} then 
processing goes to step 1 1 12 in Figure 9. 

Referring to Figure 14, an example of the processing In Figure 10 Is described below. Figure 14 shows an 
expanded view of a source image with a defect region extending from Region 1 to Region 2, wherein each region is 
composed off pixels with similar pixel values, which are distinct between regions. There are 4 line segments in SET 
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labeled H, V. D1 and 02, which are defined through PIXEL* EL at the center or Intersection of the line segments (step 
120Z). Once defined, line segment H and DZ are removed from SET because each has a side without any non-defect 
pixels, and therefore is below a minimum threshold of 4 non-defect pixels (step 1204). Next, a functional model is fitted 
and a mean square fitting error is calculated for remaining line segments V and D1 [steps 1206-1210}. In the single 

5 color channel (K=1) example of Figure 14, the mean square fitting error for line segment D1 Is lower than the mean 
square fitting error for line segment V This is due to the excessive fitting error for dissimilar non-defect pixel values 
along line segment V extending through Region 1 and Region 2. In contrast, the non-defect pixels of line segment Dl 
all lie within Region 1 and have uniform pixel values. Thus line segment D1 has a lower fitting error than line segment 
V. Consequently, at etep 1214 line segment D1 is selected, and at step 1216 PIXELg El je replaced with an estimate 

>o value based upon the non-defect pixels in line segment D1 . 

Referring to Figure 11. an alternative embodiment for estimating corrected values for PIXELs EL Is Illustrated. Cor- 
responding steps in Figure 11 have numeral designations which correspond to those in Figure 10 end will not be 
described again. Accordingly, once the mean square error calculations are made at step 1 21 0, then a weight (w h|k ) is 
calculated at step 1220 for each channel k for each line segment n in SET, where: 



is 



y-1 



20 



As shown by the equation (EQ, 13), the lower the mean square fitting error for a givei line segment in a given chan- 
nel, the greater will be the calculated weight. Next, an intermediate estimate value of PIXEL SEL (l<3L, k ) | S interpolated 
for each channel kfor each line segment in SET (etep 1222]. Once the interpolated process is completed, a final esti- 
ss mate value of PIXEL gEL 



(FGL SELk > 



30 is calculated for each channel k (step 1 224), where: 



35 



(EQ. 14) 



i-1 



Thus, the final estimate value of P IXEL^ EL is based upon a contribution of each of the calculated intermediate esti- 
40 mate values for each channel of each line segment relative to their respective weights. To illustrate this, if for example 
the line segments are those shown in Figure 14, then intermediate estimate values -for FIXE Lbfi are calculated for line 
segments V and Dl because all other line segments were removed at step 1204. But, sines line segment D1 has a 
lower mean square fitting error than line segment v; the intermediate estimate value for line segment D1 contributes 
significantly more to the final estimate value of PIXEL GEL than the intermediate estimate value for line segment V Once 
4? the estimation of values for PIXEL^ is completed (step 1216) then processing goes to step 1 1 12 in Figure 9. 

Referring back to Figure 9, once estimated values tor PIXELs EL are calculated, then at step 1112 they are stored 
in the corrected image area of memory 14. A check is then made at step 1114 to determine whether the end of the 
source image has been reached. In other words, have all pixels in the source image been scanned by the window oper- 
ator. If not, then the "no" branch is taken to step 1 1 15 where ihe window operator is advanced, and set, preferably cen- 
60 tered, to the next pixel in the image and steps 1 1 07-1 1 14 are repeated as previously described. 

Once the end of image is reached, the "yes 11 branch is taken from step 1 1 14 to step 1116, where the corrected 
image stored in memory 14 is output to the image output device 24 at step 1 1 16 and the end of the process art step 
1118. An example of the corrected image is shown in Figure 2(c), wherein the defect 1026 has been corrected and is 
no longer visually discernible. 

ss in two alternative embodiments, the size of each line segment at step 1 202 in Figures 1 0 and 1 1 is variable along 
its respective radial line through PIXEL SEL . Thus, step 1200 for setting the line segment size is no longer needed. Line 
segments are extended on both sides of PIXEL SEL . The composition of each line segment Is determined by extending 
each line segment along a vector (i.e., its respective radial line) from each side of PIXEL@ EL as described below. 
In the first alternative embodiment, line segments are separately extended on each opposite side of PIXEL 3EL on 
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a vector along Its respective radial line until each side Is composed of a certain number of non-defect pixels, ND, (not 
shown or described with respect to Figures 10 and 11). Preferably, ND represents two or three non-defect pixels. If a 
line segment cannot be extended on either side of PIXEL SEL to include ND non-defect pixels* then that line segment is 
not included in SET. This can occur because the line segment extension reached the cuter edge of the source image 
(or delect map) or a maximum distance from PIXELsel without locating ND non-defect pixels. Once both sides of any 
line segment include ND non-def act pixels, it is assured that the line segment has a proper number of non-defect pixels 
for subsequent processing. 

Referring to Figures 1 2 and 13, the second alternative embodiment will now be discussed. Figure 12 is a -flew chart 
describing the extension of one side of line segment LS n from PIXEL 6EL . Starting at step 1226, multiplier A is first set 
to 1 (step 1228). Variable Q is then set to a desired initial number of non-defect pixels from PIXEL SEL , and variable E 
to a desired Increment number of non-defect pixels (step 1230). Next, the first Q adjacent non-defect pixels for the line 
segment LS n extension from FIXEL SEL is set as Group 1 along a vector corresponding to line segment LS n radial line 
(step 1232). A series of stopping conditions for extending line segment LS n are then checked. First, if Group 1 cannot 
be established either due to the lack of Q adjacent non-defect pixels or that the edge of the source image (or defect 
map) was reached, then path 1262 from step 1232 is followed to step 1260 where line segment LS n b not included in 
SET. A check 1 234 is performed to test if Group 1 extends beyond the source image and, if rt does, extension is stopped 
(step 1255). Next a check is performed to determine whether the number of pixels in Group 1 exceeds a maximum 
threshold of line segment non-defect pixels TH MAX (step 1236). If so, then the "yes" branch is taken to step 1257 where 
the line segment LS n extension stops at the pixel in Group 1 before TH MAX was exceeded. TH^x represents the max- 
imum number of non-defect pixels any one side of PIXELq EL may be composed of. 

If the number of pixels in Group 1 dees not exceed TH^, then the "no" branch is taken to step 1238 where the 
first Q + (A*E) adjacent non-defect pixels from P IXEL SE l along the same vector as Group 1 is sat as Group 2. If Group 
2 cannot be established either due to a lack of Q + (A*E) adjacent non-defect pixels, or that the edge of the source 
image (or defect map) was reached, then path 1263 is taken to step 1261 where line segment LS„ extension slops at 
the last pixel in Group 1 . Q represents a base number of non-defect pixels in line segment LS n , and E represents the 
number of successive non-defect pixels added to that base number In each Iteration of the extension process, as will 
be shown below. 

Figure 13 illustrates an example of extending line segment LSn Along a vector 1028 from PIXEL 3EL in defect region 
1026 of source image 1025. The black dots along vector 1028 in Figure 13 represent adjacent non-defect pixels from 
PIXEL$el ( not drawn to scale). In this example, Q equals 3 and E equals 2. TTiug, Group 1 comprises the first three 
adjacent non-defect pixels closest the PIXEL 3EL , and Group 2 comprises five non-dafect pixels, i.e., the three non- 
defect pixels In Group 1 and the next two adjacent non-defect pixels from PIXEL aEL after the non-defect pixels of Group 
1. 

Next, a functional model is calculated (1236) for each channel k of the pixels in each group. Then, at step 1 240 a 
mean square fhting error (MSE ak ) is calculated for each channel k of their non-defect pixel? from their model, where g 
= 1 and 2, representing Group 1 and Group 2 respectively. Functional model and WISE calculations were described ear- 
lier in reference to Figure 10. Next, the MSE's for each channel of Bach group are summed to provide a sum mean 
square error value (SMSE fJ ) for each group g (step 1242), where: 

K 



In the preferred embodiment 8MSE B is calculated only when necessary i .e. the 6M8E 2 is stored as SMSE 1 for the 
next extension step, 

Next, a RATIO is calculated for the sum mean square errors of Group 2 to Group 1, 
RATIO = SMSE 2 /Max [SMSE t , TH a ] , where TH 2 is greater than 0 and the Max function selects the greater of the two 
terms SMSE 1f and TH 2 (step 1244). 

The next three checks are based upon SM5E 1( SMSE 2 and RATIO. The first check is performed only once at the 
beginning of the extension process when A - 1. The first check is whether the sum mean square error calculated for 
Group 1 , SMSE-, , is greater than a first threshold. TH T (step 1246). If so, then the "yes" branch is taken to step 1254 
where line segment LS n is not included ii SET THe second check is whether the sum mean square error of Group 2, 
SMSE 2 , is greater than the first threshold, TH 1 (step 1247). The third check is whether RATIO is greater than a third 
threshold, TH S (step 124B). These three thresholds TH 1p TH 2 , and TH 3 are empirically determined such that the more 
statistically consistent the values of the non-defect pixels are within each group to their model, the further line segment 
LS n will extend from PIXEL aEL . Preferably, these thresholds are selected to provide that TH f Is greater than TH a . and 
TH 3 is greater than 1. If all three checks at steps 1246, 1247, and 124B are false, then their "no ,, branches are taken to 
step 1250 where the Group 1 pixels are replaced with the pixels of Group 2. Then, A is indexed by one [step 1251) and 
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steps 1236-1246 are repeated as previously described, wherein Group 2 now Includes the next E adjacent non-defect 
pixels from PIXEL SEL in addition to the pixels of Group 1. Using Figure 13 as an example, Group 1 now has five non- 
defect pixels from P IXEL SEL and Group 2 has se*en non-defect pixels from PIXEL S£L along vector 1 028. 11 either the 
second check (step 1247) or third check (step 124$) is true, then their "yes" branch is taken to step 1253 where the line 
segment LG n extension stops at the last non-deled pixel furthest from PIXEUg EL In Group 1 . Groups 1 and 2 will con- 
tinue to be expanded in the above manner until at least one stopping condition is satisfied, leading to end 1259 

Once the extension of one side of line segment LS n from PIXEL SEL is completed, without an indication of non-inclu- 
sion in SET, the above process is repeated for the other side of line segment LS n . in this manner, the total composition 
of defect and non-defect pixels for each side of line segment LS n is determined or fixed. The above is repeated for each 
line segment through PIXEL S&L Final!/, the SET of line segments defined at step 1202 comprises the line segments 
which are extended on both sides of PIXEL SEL without a non-Inclusion Indication, 

Of the above two alternative embodiment, the second embodiment is preferred because it employe statistical anal- 
ysis of non-defect pixel values (e g., SMSE) in extending line segments, rather than a constant number of non-defect 
pixels ND. Further, in the second embodiment, those skilled in the art can appreciate other types of stopping conditions 
may also be used than those described to provide different line segment extension criteria. 

Preferably in this apparatus, once non-defect pixels have been estimated and stored in the corrected image (steps 
1110 and 1 11 2 in Figure 9), their estimated corrected values are not used in subsequent correction of other non-defect 
pixels in the source image. However, where after a single scan of the source image defect pixels remain uncorrected, 
corrected pixel values alternatively can be used in a second scan for correcting defect pixels. This may be required 
where the defect region is very large, or the defect region lies near the boundaries of the source imago. 

The preceding examples are set forth to illustrate specific embodiments of this invention and are not intended to 
limit the scope of the invention. H will be understood that variations and modifications can be effected within the spirit 
and scope of the invention. 
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Claims 

1 . A method of correcting one or more defect pixels in a defect region off a source image, which image has both the 
defect pixels and non-defect pixels each of which is represented by at least one defect pixel signal and non-defect 
pixel signal, respectively, the method comprising: 

a) searching for edges in a non-defect region of the image adjacent to the defect region by: (i) comparing rel- 
ative pixel parameters In the non-defect region with a predetermined tolerance to locate a sequence of pixels 
representing an edge segment the members of which have relative parameters within the tolerance and which 
sequence is within a predetermined distance of the defect region; and (ii) comparing the length of an edge seg- 
ment with a minimum length to identify as an edge a segment with at least the minimum length; 
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b) generating at least one pixel signal for the defect region based on the signal of at least one non-defect pixel 
lying in a direction -from the defect pixel which is determined by the direction of an edge. 

2. A method of correcting one or more defect pixels in a defect region of a source image, which image has both the 
5 defect pixels and non-defect pixels each of which Is represented by at least one defect pixel signal and non-defect 

pixel signal, respectively, the method comprising: 

a} searching for edges in a non-defect region of the image adjacent to the defect region by: (Q generating a gra- 
dient field for the non-defect region; (ii) comparing gradients of the pixels to locate a start pixel in the non-defect 

to region which has a gradient magnitude exceeding a preselected threshold; (iii) compering at least relative 

directional gradient attributes In the non-defect region with a predetermined tolerance to locate sequences of 
pixels each representing an edge segment, each sequence extending in at least one direction from a start 
pixel, the members of which have at least relative directional gradient attributes within the predetermined tol- 
erance and which sequenca is within a predetermined distance of the defect region; (iv) comparing the length 

'5 of edge segments with a minimum length to identify as edges those edge segments with at least the minimum 

length; 

b) rf at least two edges are found, comparing their relative directions and magnitudes with predetermined tol- 
erances; and 

c) generating pixel signals lor the defect region between respective ends of two edges which have relative 
?o directions and magnitudes within the predetermined tolerances, which signals are based on at least one pixel 

signal of the two edges. 

3. A method according to claim 2 wherein the non-defect region which is searched for edges is comprised of a plurality 
of local non-defect regions disposed about the defect region, each local non-defect region being searched in turn 

£5 for an edge, and wherein the start pixel for each local non-defect region is the pixel with the highest gradient mag- 
nitude In that local Image area. 

4. A method of correcting one or more defect pixels in a defect region of a source image, which image has both the 
defect pixels and non-defect pixels each of which is represented by at least one defect pixel signal and non-defect 

20 pixel signal , respectively, the method comprising: 

a) searching a non-defect region of the Image adjacent to the defect region for sequences of pixels represent- 
ing edge segments, and comparing the length of each segment with a predetermined minimum length to locate 
as edges those segments of at least the minimum length; 
35 b) searching for a geometric primitive by comparing at least one geometric parameter of a set of edges with a 

geometric constraint corresponding to a geometric primitive, a geometric primitive being detected when the set 
si least meets the geometric constraint; 

c) generating pixel signals for defect region pixels each based on the signal of at least one non-defect pixel 
lying on a predictor line having a direction determined by the direction of at least one part of a detected geo- 
40 metric primitive. 

5. A method according to claim 4 wherein: the locations of the edges together define an edg» map and wherein, prior 
to generating pixel signals, a detected geometric primitive is extracted by extending or reducing, in the map, one or 
more edges of the detected primitive so that they intersect in the same pattern as the primitive, or by replacing in 

46 the map the edges detected as a primitive with one or more new edges representing -the primitive. 

6. A method according to claim 4 wherein sequences of pixels representing edge segments are detected by: (i) gen- 
erating a gradient field for the non«Jefect region; (ii) comparing at least one relative gradient attribute of pixels in 
the non-defect region with at least one predetermined tolerance to locate a sequence of pixels the members of 

so which have at least one relative gradient attribute within the predetermined tolerance. 

7. A method according to claim 4 wherein, for at least two edges of the set which meets the geometric constraint, a 
parameter representative of pixel signals in a region adjacent to at least a first side of one edge of the set is com- 
pared with the parameter representative of pixel signals in a region adjacent to a corresponding first side of the 

5s other edge of the set, a geometric primitive being detected only if tha difference in those parameters is within a pre- 
determined tolerance. 

8. A method according to claim 7 wherein a parameter representing pixel signals in a region adjacent to a second side 
of ona edge of tha set is compared with the parameter representing pixel signals in a region adjacent to a conre- 
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spending second side of the other edge of the Get, a geometric primitive being detected only If the difference In the 
first Bide parameters and the difference in the second side parameters are both within predetermined tolerances. 

9. A method according to claim 5 wherein the signal generated for a delect pixel within a predetermined distance of a 
plurality of neighboring parts of a detected geometric primitive In the edge map, Is determined by: 

firstdetermining estimated pixel signals for the defect pixel based on at least one non-defect pixel lying on each 
of tha predictor lines passing through the defect pixel, And 

then determining a final pixel signal which is a weighted sum of estimated pixel signals, each estimated pixel 
signal being weighted in an inverse relationship to the distance between the predictor line and the correspond- 
ing geometric primitive part. 

10. A method according to claim 1 additionally comprising: 

(a) comparing the distance between a defect region pixel and a detected edge, with a threshold; 

(b) -for a defect region pixel not within the threshold, generating a pixel signal by: 

allocating a plurality of line segments through the defect pixel, wherein each said line segment is com- 
posed of the defect pixel and the non-defect pixels about said selected defect pixel ; 
determining at least one representative model of the non-defect pixel signals along each said line seg- 
ment; 

determining the deviation of the non-defect pixel signals along each said line segment from the model cor- 
responding to the line segment; 

selecting -the line segment having the lowest deviation which does not intersect a detected edge; and 
determining corrected pixel signals for said selected defect pixel baaed on at least one of the non-defect 
pixel signals In said selected line segment. 

1 1 . A method according to claim 5 additionally comprising: 

(a) comparing the distance between a defect region pixel and the nearest part of a detected primitive in tha 
edge map, with a threshold distance; 

(b) for a defect region pixel beyond the threshold distance, generating a pixel signal by: 

allocating a plurality of line segments through the defect pixel, wherein each said line segment is com- 
posed of the defect pixel and the non-defect pixels about said selected defect pixel ; 
determining at least one representative model of the non-defect pixel signals along each said line seg- 
ment; 

determining the deviation of the non-defect pixel signals along each said line segment from the model cor- 
responding to the line segment; 

selecting the line segment having the lowest deviation which does not intersect a part, in the map, of a 
detected primitive; and 

determining corrected pixel signals for said selected defect pixel based on at least one of the non-defect 
pixel signals in said selected line segment. 

12. A method of correcting at least one defect pixel in a defect region of a source image, which image has both the 
detect pixel and non-defect pixels each of which Is represented by at least one defect pixel signal and non-defect 
pixel signal, respectively, the method comprising: 

allocating a plurality of line segments through a selected defect pixel, wherein each said line segment is com- 
posed of the defect pixel and non-defect pixels about said selected defect pixel; 
determining at least one representative model of the non-defect pixel signals along each said line segment; 
determining the deviation of the non-defect pixel signals along each said line segment from the model corre- 
sponding to the line segment; 

determining at least one dominant direction of the local gradient iield; 
selecting at least one line segment based on a figure of merit; 

calculating an estimated pixel signal tor the defect pixel based on at least one non-defect pixel for each 
selected line segment; 

determining a final pixel signal which is a weighted sum of estimated pixel signals with weights proportional to 
figures of merit of the selected line segments, wherein the figure of merit is a value which increases with 
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decreased deviation ol the non-defect pixels along the selected line segment, and Increases with the alignment 
of the line segment with the dominant directions of the local gradient field. 

13. An apparatus for correcting one or more defect pixels in a defect region of a source image, which image has both 
the defect pixels and nondefect pixels each of which Is represented by at least one defect pixel signal and non- 
defect pixel signal, respectively, the apparatus comprising a programmed computer with a memory -for storing the 
source image, the programmed computer having: 

a) means for searching for edges in a non-defect region of the image adjacent to the defect region by: (i) com- 
paring relative pixel parameters in the non-defect region with a predetermined tolerance to locate a sequence 
of pixels representing an edge segment the members of which have relative parameters within the tolerance 
and which sequence is within a predetermined distance of the defect region; and (ii) comparing the length of 
an edge segment with a minimum length to identify as an edge a segment with at least the minimum length; 

b) means for generating at least one pixel signal for the defect region based on the signal of at least one non- 
defect pixel lying in a direction from the defect pixel which is determined by the direction of an edge. 

14. An apparatus for correcting one or mare defect pixels in a defect region of a source image, which image has both 
the defect pixels and non-defect pixels each of which is represented by at least one defect pixel signal and non- 
defect pixel signal, respectively, comprising a programmed computer with a memory for storing the source image, 
the programmed computer having: 

a) means for searching a non-defect region of the image adjacent to the defect region tor sequences of pixels 
representing edge segments, and comparing the length of each segment with a predetermined minimum 
length to locate as edges those segments of at least Hie minimum length; 

b) means for searching for a geometric primitive by comparing at least one geometric parameter of a set of 
edges with a geometric constraint corresponding to a geometric primitive, a geometric primitive being detected 
when the set at least meets the geometric constraint; 

c) means for generating pixel signals for defect region pixels each based on the signal of at least one non- 
defect pixel lying on a predictor line having a direction determined by the direction of at least one edge of a 
detected geometric primitive. 

15. A computer program product comprising a computer readable storage medium Including program means for cor- 
recting one or more defect pixels in a defect region of a source image, which image has both the defect pixels and 
non-defect pixels each of which is represented by at least one defect pixel signal and non-defect pixel signal, 
respectively, the program means comprising: 

a) means for searching for edges in a non-defect region of the image adjacent to tha defect region by: (i) com- 
paring relative pixel parameters in the non-defect region with a predetermined tolerance to locate a sequence 
of pixels representing an edge segment the members of which have relative parameters within the tolerance 
and which sequence is within a predetermined distance of the defect region; and (ii) comparing the length of 
an edge segment with a minimum length to identify as an edge a segment with at least tha minimum length; 
and 

b) means for generating at least one pixel signal for the defect region based on the signal of at least one non- 
defect pixel lying in a direction from the defect pixel which is determined by the direction of an edge. 

16. A computer program product, comprising a computer readable storage medium with program means for correcting 
one or more defect pixels in a defect region of a source image, which image has both the defect pixels and non- 
defect pixels each of which is represented by at least one defect pixel signal and non-defect pixel signal, respec- 
tively, the program means comprising: 

a) means for searching a non-defect region of the image adjacent to the defect ragion for sequences of pixels 
representing edge segments, and comparing the length of each segment with a predetermined minimum 
length to locate as edges those segments of at least the minimum length; 

b) means for searching for a geometric primitive by comparing at least one geometric parameter of a set of 
edges with a geometric constraint corresponding to a geometrio primitive, a geometric primitive being detected 
when the set at least meets the geometric constraint; and 

c) means for generating pixel signals for defect region pixels each based on the signal of at least one non- 
defect pixel lying on a predictor line having a direction determined by the direction of at least one edge of a 
detected geometric primitive. 



EP0 778 543 A2 





O 

CM 



" DC 



EP0 778 543 A2 




FIG. 2C 



EP0 778 543A2 




EP 0 778 543 A2 



401 



FIRST DEFECT 
NEIGHBORHOOD 



402 
VI 



I 



FIND GRADIENT MAXIMUM 
IN LOCAL NEIGHBORHOOD 
OF DEFECT SET EDGE 
START POINT 



NEW DEFECT 
NEIGHBORHOOD 



FIG. 4A 



FIG. 4B 



FIG. 4 




40 6 N T 



CALCULATE EDGE 
DIRECTION 




END 



NEXT POINT IN 
EDGE DIRECTION 





^420 


RESET TO EDGE 
STARTING POINT 
REVERSE EXPANSION 
DIRECTION 



422- 



424' 




CALCULATE AREA STATISTICS I 

1 1 



CALCULATE DEFECT ENTRY DIRECTION 



FIG. 4A 



EP 0 778 543 A2 




428- 



NEXT POINT IN DEFECT 
ENTRY DIRECTION 




CALCULATE EDGE 
DIRECTION 




TAX SIZE OR 



EXIT LENGTH 



440- 



NEXT POINT IN 
EDGE DIRECTION 




450 



FIG. 4B 



-444 

<MIN EDGE\ 
-LENGTH?^ 

f| ^446 



CALCULATE AREA 
STATISTICS 




STORE IN 
EDGE MAP AS 
CONNECTED 
EDGE 

T 



STORE IN EDGE 
MAP AS NOT 
CONNECTED 
EDGE 

— czz 



'452 



EP0 778 543A2 



502^ 



f START 

_3Z 



FIRST EDGE B. i=1 
i 



504 



503 

i 



INIT EDGE LIST LB WITH EDGE B j 



I 



ADD TO LIST LB EDGES FROM LOCAL 
NEIGHBORHOOD OF DEFECT ENTRY PART OF B : 




506 



REMOVE FROM LIST LB 
EDGES NOT MATCHED 
WITH B| 



511 



ADD TO LIST LB EDGES 
FROM LOCAL NEIGHBOR- 
HOOD OF DEFECT EXIT 
PART OF B 



SELECT BEST EDGE 
BB | FROM LIST LB 




<BBj CONNECTED?: 



513 



CREATE LIST LC OF EDGES 
p^I FROM LOCAL NEIGHBORHOOD 
OF DEFECT EXIT PART OF SB 



5.14 



516 




515 



CONNECT B9. . EXTRACT LINE 



J 



REMOVE FROM EDGE MAP ALL EDGES ON LIST LB BUT BB . 



518s 



NEXT EDGE 
B| 




FIG. 5A 



EP0 778 543 A2 



FIG5B1 




FIQ5B2 


FIG. SB 



(START) 




FIRST POINT P. OF EDGE BB , J-1 




566 



NEXT 


EDGE POINT 






J-J+1 


564 








J 





563 

CONNECTED? 




584- 



REMOVE 
BOTH 
EDGES 



5$2~ 



EXTRACT 
CORNER 



570 



1 



REMOVE PART OF BB( 
FROM Pj TO LAST POINT 



592 



1 



59Q« 



NEXT EDGE BB ■ 1=1+1 



LAST EDGE? 



EP0 778 543 A2 



528 




530^ 

p ANY \ t 

—<i INTERSECTED^ 
EDQES^AJ,»Pr? 





^540 


CHECK Gl 
PRIMITIVE Cl 


EOMETRIC 
DNSTRAINTS 




CHECK GEOMETRIC 
PRIMITIVE CONSTRAINTS 




FIG. 5B2 



EP0 778 543 A2 





EP0 778 543A2 




702 



FIG.7 



(START] 



FIG.7A 



FIRST DEFECT PIXEL. U1~| 

T 



704 



CREATE N LINE SEGMENTS THROUGH DEFECT PIXEL P 
CREATE LIST OF NEIGHBORING PARTS NPi j AND 
CORRESPONDING DISTANCES D|,j j-1,..,J 1 



708 




710 



FIND MIN DISTANCE DMIN 



760- 



CREATE N LINE SEGMENTS THROUGH 
DEFECT PIXEL P } 



762- 



NEXT SEGMENT, n-1 




776^ 



CALCULATE FIGURE OF MERIT F- 



NEXT SEGMENT 

n=n+1 



784 



WEIGHTED AVERAGING 
OF NON-DEFECT PIXELS 



M 



U;?i W i.n E i,n,k 
k-i,..,K 




L 



SELECT M 
BEST SEGMENTS 



780 



EP0 778 543A2 



t^^jpeT^-^f, 
'DI5ECTI9N?' 



716 



Pl^ =0 k»l,..,k Wj-0 



714 



720 



718 



FIRST NEIGHBORING PART.j=1 



DETERMINE PREDICTOR FOR NEIGHBOR- 
ING PART NP, 




ESTIMATE PIXEL VALUES E |j k k-1,..,K 




.728 



P U =P| (k + 57, E M-k k=1,..,K 



730 

V 



1 



Wj =«Wj +_L 




NEXT NEIGHBORING 
PART. H+1 



736 
1/ 



740 



2. 



P i.k - k-1,..,K 
W l 



794* 



3 



J 



742 



STORE PIXEL VALUES! " 790 



NEXT DEFECT 
PIXEL. 




EP0 778 543A2 





FIG. 8d FIG 8e FIG. 8f 



EP0 778 543A2 



START L-1098 




READ AND STORE IMAGE 



■ 1100 



FIG.9 



1115 



ADVANCE 
WINDOW 
OPERATOR 
TO NEXT 
PIXEL 



LOCATE DEFECT REGIONS 
JN IMAGE 



I 



^1102 



BUILD AND STORE DEFECT MAP L-1104 

* 



SET WINDOW OPERATOR 110e 
TO 1ST PIXEL ^ 



1107 




SELECT DEFECT PIXEL 
(PIXELsel ) 



-1108 



ESTIMATE VALUES 
FOR PIXEL sel 



-1110 



,1109 



STORE 
PIXEL 
VALUE IN 
CORRECTED 
IMAGE 



STORE ESTIMATED VALUES FOR 
PIXEL SEL IN CORRECTED IMAGE 




1112 



OUTPUT /*™*\ 
CORRECTED — {END! 
IMAGE >- / 
1118 



1116 



EP 0 778 543 A2 



FIG.1QA 



FIQ.10B 



FIG.10 



(sTAR^1198 



SET LINE SEGMENT SIZE TO X 



DEFINE SET OF LINE SEGMENTS 
THROUGH PIXEL SEL 



.1200 



1202 



1204 



REMOVE FROM SET ANY LINE SEGMENT HAVING 
LESS THAN A MINIMUM THRESHOLD OF NON- 
DEFECT PIXELS ON EACH SIDE OF PIXEL gEL 



INCREASE 

LINE 
SEGMENT 

SIZE 
X BY Y 



.1211 



N 



,1206 

IS THE REMAINING" 
I LIMBER OF LINE SEGMENTS^ 
N>0? 



REPLACE ALL REMOVED 
LINE SEGMENTS IN SET 



_~1207 




FIG. 10A 



CALCULATED ESTIMATE VALUE FOR 

EACH CHANNEL k OF PIXELSEL 
BASED ON WEIGHTED AVERAGE OF 
NON-DEFECT PIXELS IN SET 



EP0 778 543A2 



J ^.1208 



CALCULATE A FUNCTIONAL MODEL FOR EACH CHANNEL 
k OF EACH LINE SEGMENT n IN SET 



T 



.1210 



CALCULATE A MEAN SQUARE FITTING ERROR fMSEn.k) 
OF THE NON-DEFECT PIXELS FROM THE FUNCTIONAL MODEL 
FOR EACH CHANNEL k OF EACH LINE SEGMENT n IN SET 



-1212 



CALCULATE FOR EACH LINE SEGMENT n A TOTAL MEAN 
SQUARE FITTING ERROR (MSE tota | ) WHERE 

M5E TOTAL n --f£M8E„ ik 



I 



1214- SELECT THE LINE SEGMENT WITH LOWEST MSETOTAL 



T 



INTERPOLATE USING AT LEAST 2 NON-DEFECT PIXELS OF 
THE SELECTED LINE SEGMENT ON EITHER SIDE OF PIXEL S EL 
TO DETERMINE AN ESTIMATE VALUE FOR EACH 
CHANNEL k OF PIXEL SEL 




^1216 




1218 



FIG. 10B 



EP0 778 543 A2 







FIG.11A 


FIG.11B 



STARTL-1198 



FIG.11 




SET LINE SEGMENT SIZE TO X 



1200 



DEFINE SET OF LINE SEGMENTS 
THROUGH PIXEL SEL 



1202 



1204 



REMOVE FROM SET ANY LINE SEGMENT HAVING 
LESS THAN A MINIMUM THRESHOLD OF NON- 
DEFECT PIXELS ON EACH SIDE OF PIXEL SEL 



INCREASE 

LINE 
SEGMENT 

SIZE 
X BY Y 



1211 



N 



.1206 

IS THE REMAINING" 
I LIMBER OF LINE SEGMENT! 
N>0? 



REPLACE ALL REMOVED 
LINE SEGMENTS IN SET 



-1207 




FIG. 11A 



CALCULATED ESTIMATE VALUE FOR 

EACH CHANNEL k OF PIXELSEL 
BASED ON WEIGHTED AVERAGE OF 
NON-DEFECT PIXELS IN SET 



EP0 778 543 A2 



1208- 



1210 



CALCULATE A FUNCTIONAL MODEL FOR EACH CHANNEL 
k OF EACH LINE SEGMENT n IN SET 



CALCULATE A MEAN SQUARE FITTING ERROR (MSEn.k ) 

OF THE NON-DEFECT PIXELS FROM THE FUNCTIONAL MODEL 
FOR EACH CHANNEL k OF EACH LINE SEGMENT n IN SET 



1220 



1222 



CALCULATE A WEIGHT (W n |<) FOR EACH CHANNEL k 
FOR EACH LINE SEGMENT D IN SET, WHERE: 

w n,k-i--N — 

ki 



X 



INTERPOLATE AN INTERMEDIATE ESTIMATE VALUE OF 

PIXEL SEL(IGLn,k) FOR EACH CHANNEL k 
FOR EACH LINE SEGMENT n IN SET 



1224 



A. 



CALCULATE A FINAL ESTIMATE VALUE OF 
PIXEL SEL (FGLSELk> FOR EACH CHANNEL k. WHERE; 

FGL SEL k u| 

N 

C W i.k 
j-1 



1218 r^x 

^endJ 



FIG. 11B 



EP0 778 543A2 




STARTL.1226 




1228 



FIG. 12 



1260 

i 



SET Q TO INITIAL NUMBER 
OF NON-DEFECT PIXELS, AND 
E TO INCREMENT NUMBER OF 
NON-DEFECT PIXELS 



1230 



DO NOT 
INCLUDE 
lLSpIN SET 



1262 



GROUP 1 - FIRST Q ADJACENT 
NON-DEFECT PIXELS IN LS n 



EXTENSION FROM PIXEL 



SEL 



1255-1 



STOP LS n EXTENSION 



BEYOND SOURCE 
IMAGE 




1232 



DOES GROUP 1 



TO PIXEL IN GROUP ip-<^XTENDE_D BEYOND' 
BEFORE EXTENDING 



SOURCE IMAG£? 
1234"^f 



1257^ 



STOP LS n EXTENSION 

TO PIXEL IN GROUP 
BEFORE THMAX 
WAS EXCEEDED 




fHE NUMBI 



A-A+1 
7" 



1251 



IS THE NUMBER^. 
OF PIXELS IN^-* 
GROU5J>TH MAX ? 

^ 1236 



1261 -J 



STOP LSn 

EXTENSION TO LAST 
PIXEL IN GROUP 1 



1263 



GROUP 2-FIRST Q+{AxE) 
ADJACENT NON-DEFECT 
PIXELS IN LSp EXTENSION 
FROM PIXELsel 

""7 

1238 



FIG.12A 



EP0 778 543A2 



1239 



CALCULATE FUNCTIONAL MODEL FOR EACH 
CHANNEL k OF THE NON-DEFECT PIXELS IN 
EACH GROUP g, WHERE s«*1 TO 2 



1240 



CALCULATE MEAN SQUARE FITTING ERROR 
[MSEgk! OF THE PIXELS FROM THE FUNCTIONAL 
MODEL FOR EACH CHANNEL k FOR EACH GROUP g 



1242 



CALCULATE SUM SQUARE ERROR FOR EACH 
GROUP [SMSE Q ] WHERE: 
K 

SMSEg- £MSE g(k 
k»1 



1244 



I 



CALCULATE RATIO-SMSEg/MAX [SMSE 1 ,TH 2 ] 



1254n 



DO NOT 
INCLUDE 
LSn in 
SET 



1253 



(end)~- 

1259 



2L 



STOP LSn 
EXTENSION 
TO LAST PIXEL 
IN GROUP 1 




SET 
GROUP 1 

TO 
GROUP 2 



1250* 



FIG. 12B 



EP 0 778 543 A2 



1025 




GROUP 1 \ 
1 023 



25 



FIG.I3 



. II. ■ LUJI I I Ml I I II I I III II I I I H I l, ,,U 



iHibiinMUHUHiitiiHrM 




region i-:v::vXv::v::-^ 





iYjViNYiVV "tV'TA'i"''' 

DEFECT 
REGION 



□ - PIXEL 
■ - PIXELsel 



FIG. 14 



